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Abstract 

Our  unaided  eyes  can  detect  two  of  the  characteristics  of  image-forming  visible  light  from  a 
scene,  namely,  the  intensity  and  the  wavelength,  which  can  then  be  encoded  into  perceptual  qualities  of 
brightness  and  color.  Our  eyes,  however,  are  effectively  blind  to  the  third  characteristic  of  light,  its 
polarization.  It  is  well  known,  however,  that  several  species  of  animals  have  visual  systems  capable  of 
detecting  light’s  polarization  and  using  the  information  so  extracted.  Understanding  the  biophysical 
mechanism  behind  the  polarization  vision  and  reverse  engineering  its  functionality  and  utility 
leads  to  exciting  novel  methods  and  techniques  in  sensing  and  imaging  with  various  applications. 
Motivated  and  inspired  by  the  features  of  polarization-sensitive  visual  systems  in  nature,  in  our 
group  we  have  been  developing  various  man-made,  non-invasive  imaging  methodologies, 
sensing  schemes,  camera  systems,  and  visualization  and  display  schemes  that  have  shown 
exciting  and  promising  outcomes  with  useful  applications  in  system  designs.  These  techniques 
provide  better  target  detection,  enhanced  visibility  in  otherwise  low-contrast  conditions,  longer 
detection  range  in  optically  scattering  media,  polarization-sensitive  adaptation  based  on  changing 
environments,  surface  deformation-variation  detection  (e.g.,  detection  of  finger  prints  on  a 
smooth  surface  using  polarization-based  vision),  “seeing”  objects  in  shadows,  and  other  novel 
outcomes  and  applications.  During  our  efforts  supported  by  this  grant,  we  have  investiagted 
various  exciting  possibilities  of  bio-inspired  polarization  imaging  with  numerous  potential 
applications.  Our  findings  are  described  in  the  attached  published  papers. 
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Adaptive  algorithms  for  two-channel  polarization 
sensing  under  various  polarization  statistics  with 
nonuniform  distributions 


Konstantin  M.  Yemelyanov,  Shih-Schon  Lin,  Edward  N.  Pugh,  Jr.,  and  Nader  Engheta 


The  polarization  of  light  carries  much  useful  information  about  the  environment.  Biological  studies  have 
shown  that  some  animal  species  use  polarization  information  for  navigation  and  other  purposes.  It  has 
been  previously  shown  that  a  bioinspired  polarization-difference  imaging  (PDI)  technique  can  facilitate 
detection  and  feature  extraction  of  targets  in  scattering  media.  It  has  also  been  established  [J.  Opt.  Soc. 
Am.  A  15,  359  (1998)]  that  polarization  sum  and  polarization  difference  are  the  optimum  pair  of  linear 
combinations  of  images  taken  through  two  orthogonally  oriented  linear  polarizers  of  a  scene  having  a 
uniform  distribution  of  polarization  directions.  However,  in  many  real  environments  the  scene  has  a 
nonuniform  distribution  of  polarization  directions.  Using  principal  component  analysis  of  the  polarization 
statistics  of  the  scene,  we  develop  a  method  to  determine  the  two  optimum  information  channels  with 
unequal  weighting  coefficients  that  can  be  formed  as  linear  combinations  of  the  images  of  a  scene  taken 
through  a  pair  of  linear  polarizers  not  constrained  to  the  horizontal  and  vertical  directions  of  the  scene. 
We  determine  the  optimal  orientations  of  linear  polarization  filters  that  enhance  separation  of  a  target 
from  the  background,  where  the  target  is  defined  as  an  area  with  distinct  polarization  characteristics  as 
compared  to  the  background.  Experimental  results  confirm  that  in  most  situations  adaptive  PDI  out¬ 
performs  conventional  PDI  with  fixed  channels.  ©  2006  Optical  Society  of  America 
OCIS  codes:  260.5430,  110.2970,  330.1880,  100.2960,  330.7320. 


1.  Introduction 

Polarization  is  an  intrinsic  feature  of  light  that  pro¬ 
vides  valuable  information  about  a  scene  beyond  that 
provided  by  the  scene’s  spectral  (color)  and  intensity 
distributions.  Polarized  light  has  been  studied  exten¬ 
sively  since  Fresnel’s  investigations  of  the  wave 
theory,  and  is  important  in  many  areas  of  modern 
technology  (see,  e.g.,  Refs.  1-4).  Polarization  informa¬ 
tion  has  proven  useful  in  several  fields,  including 
computer  vision,5  6  target  detection,  particularly  in  im¬ 
aging  targets  in  scattering  media,  such  as  water,  fog, 
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etc.,  in  feature  extraction,7"22  and  in  material  classi¬ 
fication.2324  Polarization  parameters,  e.g.,  Stokes  pa¬ 
rameters,  are  in  general  more  sensitive  to  the  nature  of 
a  scattering  surface  than  the  total  intensity  is.  In  such 
a  case,  polarization  imaging  techniques  offer  the  pos¬ 
sibility  of  producing  images  with  higher  inherent  vi¬ 
sual  contrast  than  conventional  image  processing  of 
the  intensity  distribution.3 

The  polarization  of  light  is  not  discernible  to  the  un¬ 
aided  human  eyes,25  but  polarization  has  been  shown 
to  provide  valuable  information  to  other  species.  In 
1949,  Nobel  laureate  Karl  von  Frisch  established  that 
honeybees  through  their  perception  of  polarized  light 
use  the  Sun  as  a  compass  through  their  perception  of 
the  polarization  pattern  of  light  scattered  from  the 
sky.26’27  After  von  Frisch’s  discovery,  other  research¬ 
ers  began  to  investigate  polarization  vision  and  found 
it  in  many  different  species,  including  amphibians, 
arthropods,  desert  ants,  octopuses,  and  probably  fish 
(see,  e.g.,  Refs.  28-41).  These  animals  use  polariza¬ 
tion  information  in  many  different  ways,  e.g.,  navi¬ 
gation,  detecting  water  surfaces,  enhancing  visibility 
(similar  to  colors),  and  perhaps  even  for  mutual  com¬ 
munication. 

From  its  utilization  by  animals,  it  is  clear  that  the 
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pattern  of  polarization  in  an  image  of  a  scene  is  a 
potentially  rich  source  of  information.  While  the  hu¬ 
man  eye  is  “polarization  blind,”  man-made  imaging 
systems  have  been  developed  to  collect  polarization 
information  from  scenes.5  6  An  important  issue  for 
such  systems  is  how  to  process  and  display  the  po¬ 
larization  information  after  it  is  collected  by  the  im¬ 
aging  system  to  enhance  our  vision  of  the  scene. 
Inspired  by  polarization  vision  of  certain  animal  spe¬ 
cies,  in  earlier  studies,  our  group  introduced,12’42’43 
polarization-difference  imaging  (PDI)  processing.44 
We  demonstrated  that  optical  imaging  systems  uti¬ 
lizing  PDI  techniques  may  help  the  detection  of  tar¬ 
gets  in  scattering  media  even  when  the  fraction  of  the 
light  is  polarized  by  only  a  few  percent,  and  that  such 
enhancement  can  increase  by  up  to  threefold  the  dis¬ 
tance  over  which  targets  can  be  reliably  detected  near 
threshold  visibility.12  42’43  We  have  also  investigated 
the  issue  of  optimal  representation  of  polarization 
information  for  the  polar-blind  human  eye.43-45 

The  idea  that  polarization  vision  may  involve  adap¬ 
tation  to  the  environmental  polarization  is  bioinspired 
as  well.  Insects  employ  a  retinal  filter  consisting  of  an 
array  of  ommatidia  (polarization-sensitive  photorecep¬ 
tors)  that  is  approximately  matched  to  the  polarization 
pattern  of  the  sky.32  This  filter  works  as  follows:  The 
summed  output  from  all  the  polarization  analyzers  of 
the  polarization-sensitive  area  reaches  a  maximum 
when  a  match  between  the  receptor  array  and  the 
celestial  pattern  has  been  approximately  achieved. 
This  provides  the  insect  with  the  information  of  how  to 
align  its  longitudinal  body  axis  with  the  symmetry 
plane  of  the  sky.  To  find  a  proper  direction,  the  insect 
has  to  change  its  orientation  and  perform  a  check  of 
the  polarization  pattern  for  each  angle  of  orienta¬ 
tion.46’47 

Inspired  by  the  concept  of  matched  filters3132  we 
develop  a  polarization  imaging  technique  based  on 
PDI,  but  that  is  adaptable  to  the  environmental  con¬ 
ditions,  i.e.,  to  the  polarization  background  of  a  scene. 
The  proposed  polarization-based  system  adjusts  itself 
to  enhance  the  segregation  of  targets  from  the  back¬ 
ground  in  a  manner  dependent  on  the  polarization 
statistics  of  the  scene. 

The  paper  is  organized  as  follows:  in  Section  2,  we 
give  a  brief  overview  of  the  polarization  concept  and 
define  the  polarization  parameters  used  in  the  paper. 
In  Section  3,  we  describe  the  adaptive  polarization- 
difference  imaging  (APDI)  algorithm.  Section  4  is  de¬ 
voted  to  the  validation  of  the  proposed  technique, 
employing  both  experiments  and  simulations.  In  Sec¬ 
tion  4,  we  illustrate  the  performance  of  the  algorithm 
in  target-against-background  detection  using  the  ex¬ 
perimental  data  obtained  in  a  natural  environment. 
In  Section  5,  we  apply  the  APDI  algorithm  to  a  scene 
taken  under  natural  illumination  conditions.  In  Sec¬ 
tion  6,  we  propose  an  APDI-based  method  that  may 
be  used  in  surveillance  systems,  compare  APDI  with 
several  other  methods  in  polarization-based  imaging, 
and  discuss  the  performance  of  the  algorithm.  Finally 
in  Section  7,  we  present  our  conclusions. 


2.  Physics  of  Polarization  Imaging 

The  polarization  of  the  local  field  of  a  monochromatic 
coherent  light  source  can  be  represented  as  a  super¬ 
position  of  two  mutually  orthogonal  wave  compo¬ 
nents.  A  phase  difference  between  these  components 
produces  a  linearly,  elliptically,  or  circularly  polar¬ 
ized  wave  with  its  polarization  direction  determined 
by  the  relative  strengths  of  the  components.  Since 
human  eyes  and  most  conventional  camera  sensors 
can  only  detect  the  total  light  intensity  collected 
at  each  pixel  during  an  exposure  time  many  times 
longer  than  the  oscillation  frequency,  when  the  light 
energy  is  uniformly  distributed  over  all  polarization 
directions,  the  detected  signal  will  be  the  same  for 
any  polarization  direction.  Such  light  is  called  unpo¬ 
larized  and  emitted  by  most  common  light  sources, 
including  the  Sun  or  man-made  incandescent  light 
sources.  The  local  polarization  signal  at  the  surface  of 
a  sensor  (which  cannot  detect  phase  information)  can 
be  described  as  a  combination  of  unpolarized  and 
completely  linearly  polarized  components. 

Assuming  that  a  perfect  linear  polarizer  is  placed 
in  front  of  a  normal  polarization-insensitive  device, 
such  as  a  CCD  or  film  camera,  the  observed  intensity 
I(x,  y,  cp)  at  the  pixel  located  at  (x,  y )  is  in  general  a 
function  of  the  angle  cp  that  polarization  analyzer 
makes  with  a  reference  direction,  and  can  be  de¬ 
scribed  as 

I(x,  y,  cp )=Iu(x,  y)(  1  +  p(x ,  y)cos{2[0(x,  y)  -  cp]}), 

(1) 

where  Iv  is  half  of  the  total  pixel  intensity,  and 
p  is  the  degree  of  linear  polarization  defined  as 
[^max(*^?y)  3^)l/[^max(*^?  TO  3^)1?  where 

Imax(x,y)  and  Imin(x,y)  denote  at  each  pixel  (: x,y ),  re¬ 
spectively,  the  maximum  and  minimum  observed  in¬ 
tensity  within  a  full  rotation  of  the  analyzer.  (We  note 
that  this  definition  of  degree  of  linear  polarization, 
which  is  more  suitable  for  wideband  signals  used 
here,  is  different  from  what  is  used  as  the  degree  of 
polarization  in  the  context  of  the  Stokes  parame¬ 
ters.49  Throughout  this  paper,  we  use  the  horizon  as 
the  reference  direction,  and  angles  increase  counter¬ 
clockwise  relative  to  the  direction  of  the  horizon. 

As  is  evident  from  Eq.  (1),  at  each  pixel  of  the  image 
of  a  scene,  the  polarization  (and  intensity)  of  the  im¬ 
pinging  light  is  characterized  by  three  independent 
parameters:  thus  to  obtain  complete  information 
about  the  polarization  features  of  the  object,  at  least 
three  measurements  of  light  intensity  at  different 
angles  cp  are  required.  These  measurements  can  be 
made  either  simultaneously  by  three  CCD  cameras, 
such  that  each  camera  has  a  fixed  polarizer  set  at  a 
different  angle,  or,  assuming  the  scene  is  stable  over 
time,  by  one  camera,  taking  images  sequentially 
through  a  polarizer  oriented  at  three  different  angles. 
Our  computational  algorithm  can  be  used  with  either 
measurement  method  after  proper  calibration. 

Consider  three  images  of  a  scene,  I0,  /45,  /90,  corre¬ 
sponding  to  three  angles  of  orientation  of  the  linear 
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polarizer,  namely,  cp  =  0°,  45°,  and  90°  with  respect  to 
the  reference  direction.  From  these  three  images,  we 
can  calculate  Iu,  p,  and  0  at  each  image  point  as 
follows: 


I0(x,  y )  +  I90(x,  y) 

iu(x,  y)  = - o - 


p(x,  y )  = 


JT  h5(x,  y)~ 

2 

i 

'  ho(x,  y)~ 

21 

{[  lu(x,  y) 

1 

Iu(x,  y ) 

1 

0(x,  y)  = 


1  _J  Iu(x,  y)-I0(x,  y) 

2  tan  {ljj(x,  y)  -  /45(x,  y) 


(2) 


Once  the  parameters  Iv,  p ,  and  0  are  computed  for 
each  pixel  of  the  scene  image,  one  can  reconstruct  the 
image  intensity  that  would  be  observed  for  any  angle 
cp  using  Eq.  (1),  even  though  no  actual  pictures  are 
taken  with  a  polarizer  oriented  at  cp. 

3.  Concept  of  Adaptive  Polarization- Difference  Imaging 

A.  Polarization-Difference  Imaging 

The  concept  of  PDI  was  introduced  in  earlier  inves¬ 
tigations  of  our  research  group.12’42-45  This  idea  has 
since  been  utilized  by  other  research  groups  (see, 
e.g.,  Refs.  13,  18,  20,  and  49-51).  The  original  PDI 
system  captures  images  of  a  scene  at  two  orthogonal 
linear  polarizations.  Thus  one  obtains  a  pair  of  im¬ 
ages,  i.e.,  I0(x,  y)  and  I90(x,y)  taken  at  0°  and  90° 
orientation  of  the  polarizer,  respectively.  The 
polarization-sum  (PS)  and  polarization-difference  (PD) 
images  are  linear  combinations  of  the  intensity  images 
for  the  two  orthogonal  polarizations;  thus  given  IQ(x,  y) 
and  I90(x,  y)  one  computes: 


'PS(*,  y) ' 

1  1 

790(x,  y) 

PD(x,  y) 

1  -1 

Jo(x,  y) 

T 


where  T  identifies  the  transformation  matrix.  For  an 
ideal  linear  polarizer,  the  PS  image  is  equivalent  to  a 
conventional  intensity  image. 

Tyo52  has  shown  that  the  two  image  channels  PS 
and  PD  given  by  Eq.  (3)  are  the  principal  components 
(PCs)  of  a  scene  in  which  the  polarization  angle  has  a 
uniform  distribution;  thus  in  such  cases,  PS  and  PD 
are  optimal  channels  in  the  information-theoretic 
sense  of  carrying  maximally  uncorrelated  informa¬ 
tion  about  the  scene.  Tyo’s  ideas  were  developed  in 
analogy  with  a  principal  component  analysis  (PCA) 
of  trichromatic  (three-cone  pigment)  color  vision  by 
Buchsbaum  and  Gottschalk.53  In  Tyo’s  analysis,  as  in 
that  by  Buchsbaum  and  Gottschalk,53  the  transfor¬ 
mation  matrix  T  in  Eq.  (3)  is  derived  by  applying  PCA 
to  the  covariance  matrix  of  the  input  channels  [e.g.,  I0 
and  I90  in  Eq.  (3)]  for  a  broadband  distribution  of 
the  relevant  property  (polarization,  spectral  distribu¬ 
tions)  over  the  ensemble  of  scenes.  In  these  analyses, 
the  principal  component  with  the  largest  eigenvalue 


is  a  same-signed  sum  of  the  input  channels,  while  the 
remaining  components  are  the  “opponent,”  involving 
opposite-signed  weighting  coefficients  in  the  trans¬ 
mission  matrix.52’53  An  interesting  corollary  to  the 
optimality  of  PCA  channels  in  information  encoding 
is  that  the  opponent  channels  can  also  be  understood 
to  be  optimized  for  the  detection  of  change  relative  to 
the  average  scene  to  which  the  first  principal  compo¬ 
nent  is  tuned;  thus  the  opponent  channel(s)  in  effect 
perform  a  common-mode  rejection  of  the  statistically 
average  scene  or  background  signal,  and  in  doing  so 
enhance  the  signal-to-noise  ratio  (SNR),  and  thus  the 
detectability  by  the  opponent  channels  of  targets  in  a 
scene  that  differs  in  polarization  statistics  from  the 
background.  This  duality  in  the  sense  of  optimality  of 
PCA-derived  channels  led  us  to  seek  the  optimum 
channels  for  an  arbitrary  background  scene. 

B.  Generalization  of  Polarized-Difference  Imaging  to 
Nonuniform  Polarization  Statistics 

In  real  scenes,  both  the  polarization  orientation  and 
the  degree  of  linear  polarization  of  the  scene  have  non- 
uniform  distributions.  In  such  situations,  the  PD  sig¬ 
nal  of  Eq.  (6)  may  not  be  the  most  useful  signal  for 
detecting  targets  against  the  average  background. 
What  would  then  be  the  appropriate  image  (i.e.,  a  com¬ 
bination  of  signals)  to  reveal  hidden  polarization  fea¬ 
tures  of  the  scene  and  improve  the  target  detection? 
Answering  this  question  is  the  goal  of  this  study. 

Our  approach  to  this  goal  has  been  to  enable  our 
imaging  system  to  adapt  to  the  polarization  statistics 
of  the  background,  so  that  if  any  changes  occur  in  the 
scene  they  will  pop  out  on  one  of  the  channels.  The 
adaptation  of  the  system  to  the  polarization  statistics 
of  the  background  in  effect  performs  a  common-mode 
rejection  of  the  background.  In  such  an  approach,  it  is 
assumed  here  that  the  system  measures  the  polar¬ 
ization  statistics  of  the  scene  in  the  two  different 
stages:  once  when  only  the  background  is  present, 
and  again  when  a  target  is  present  with  the  back¬ 
ground.  Furthermore,  in  describing  the  ideal  behav¬ 
ior  of  such  a  system,  the  target  is  considered  as  a 
perturbation  of  the  background  scene,  i.e.,  a  minor 
change  that  does  not  alter  the  overall  polarization 
statistics  (Subsection  4.C). 

Consider  the  general  case  such  that  the  probability 
density  function  of  the  polarization  angle  over  the 
pixels  is  arbitrary.  Assume  that  images  of  the  scene 
with  M  XN  pixels  have  been  taken  with  two  different 
orientations  (not  necessarily  orthogonal)  of  the  polar¬ 
izer,  i.e.,  Ii  =  /(cpi)  and  lx  =  /( cp2).  Here,  we  examine 
these  two  signals  using  the  PCA  technique.55  Accord¬ 
ing  to  PCA,  the  covariance  matrix  for  such  an  arbi¬ 
trary  pair  of  images  is  defined  as 


C( cp1?  cp2)  = 


E(hh)-E?(I  i) 


E  (I  J 2)  E(Ii)E(I2) 
E(I2I2)-E\I2)  } 

(4) 


where  E(W)  =  1/M/V2m=i  2^=i  W(xm9  yn)  is  the  mean 
value  taken  over  the  ensemble  of  pixels  in  the  images, 
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and  W  is  equal  to  Il9  /2,  or  a  product  thereof.  Once  the 
eigenvalues  (X1?  X2)  and  the  eigenvectors  of  the  co- 
variance  matrix  C  are  determined,  the  transforma¬ 
tion  matrix,  which  has  the  eigenvectors  as  its  rows,  is 
formed  as  follows: 


T(<Pu  92)  = 


-P(<Pi,  92) 
«(9i,  92) 


a(9i,  92) 
P(9i,  92) 


(5) 


The  signs  of  the  scalars  a  and  (3  may  be  either  positive 
or  negative.  The  transformation  matrix  is  formed  in 
such  a  way  that  the  first  eigenvector  corresponds  to 
the  largest  eigenvalue.  The  principal  component  im¬ 
ages  are  then  computed  as  follows: 


PC1(91,  92) 
PC2(9i,  92) 


=  T(9!,  92) 


7(9i)' 

/(92)_ 


(6) 


By  analogy  with  the  transformation  matrix  of  Tyo’s 
conventional55  PDI  system,  we  consider  PCX  the 
adaptive  analog  of  the  PS  signal,  and  the  PC2  the 
analog  of  the  PD  or  opponent  signal.  We  surmise  that 
by  analyzing  the  PC2  image  of  a  scene,  certain  im¬ 
portant  features,  e.g.,  the  detection  of  a  target  in  the 
presence  of  a  standard  background  will  be  enhanced. 

All  the  parameters  in  Eqs.  (5)  and  (6)  are  functions 
of  the  two  polarizer  angles,  which  in  general,  need  not 
be  orthogonal.  For  the  case  of  a  uniform  distribution 
of  polarization  ellipse  orientations  over  the  scene,  Eq. 
(6)  reduces  to  Eq.  (3).  Our  goal  is  to  identify  a  pair  of 
analyzer  angles  and  a  pair  of  weighting  coefficients 
in  Eq.  (6)  [i.e.,  the  components  of  the  eigenvectors 
shown  in  Eq.  (5)],  which  yield  a  PC2  image  that  op¬ 
timally  enhances  the  visualization  of  novel  targets  in 
a  specific  background  scene.  The  term  “optimum” 
here  is  used  by  analogy  with  the  approach  previously 
used  in  the  analyses  of  color  vision  and  polarization 
vision,  as  described  in  Section  3. A.  In  those  analyses 
a  formal  assumption  was  made  about  the  properties 
of  the  ensemble  distribution  of  signals.  In  the  present 
analysis,  the  role  of  the  ensemble  distribution  is 
played  by  the  polarization  distribution  of  a  specific 
background  scene,  and  so  we  will  put  the  adjective 
“optimum”  in  quotations,  with  the  understanding 
that  it  is  our  task  to  demonstrate  empirically  that  the 
method  indeed  yields  a  practical  optimum  separation 
of  targets  from  the  specific  background. 

C.  Finding  a  Transformation  Matrix  to  Adapt  the 
Polarization-Difference  Imaging  System  to  a 
Specific  Background 

To  adapt  the  PDI  system  to  a  specific  background 
scene,  we  first  obtain  complete  polarization  informa¬ 
tion  on  the  background.  Specifically,  we  capture  three 
images  of  the  background  scene  for  three  orientations 
of  the  polarizer  and  then  compute  the  polarization 
parameters  that  completely  characterize  the  scene 
with  Eq.  (2).  From  these  results,  we  can  synthesize 
images  of  the  scene  corresponding  to  any  angle  of 
orientation  9  of  the  polarizer  with  Eq.  (1).  It  bears 
emphasis  that  the  derivation  of  images  correspond¬ 


ing  to  various  polarizer  orientations  9  from  the  initial 
set  of  three  images  is  not  linear. 

The  next  step  is  to  perform  the  principal  components 
analysis  as  described  in  Subsection  3.B  for  a  wide  se¬ 
lection  of  pairs  of  angles  91?  92,  deriving  the  transfor¬ 
mation  matrix  TBG(9!,  92)  of  Eq.  (6)  for  the  background 
for  all  pairs  of  angles.  This  yields  a  table  or  map  of 
the  derived  polarization  parameters  over  the  2D  space 
of  angles:  i.e.,  a^,  92),  (3(9!,  92),  X.i(9i,  92),  and 
X2(9i,  92)  for  0  <  91?  92  <  180°.  It  is  reasonable  to 
expect  that  the  “optimum”  pair  of  angles  will  be  that 
with  corresponding  extreme  eigenvalues.  When  the 
“optimum”  pair  of  angles  is  chosen,  then  Eq.  (9)  is 
applied  to  the  target  scene.  Here  we  consider,  in  par¬ 
ticular,  the  following  possible  choices  for  the  “opti¬ 
mum”  pair  (91?  92)  of  polarizer  orientations: 

Case  1.  The  pair  that  maximizes  the  eigenvector 
component  a  (and  it  minimizes  the  magnitude  of  (3). 

Case  2.  The  pair  that  maximizes-minimizes  the 
eigenvalues. 

Case  3.  The  pair  of  orthogonal  angles  that  has  the 
preferential  angle  of  the  background  polarization  as 
its  bisector. 

Consideration  of  case  3  will  allow  a  comparison  be¬ 
tween  the  conventional  PD  algorithm  and  the  new 
adaptive  algorithm  here.  In  this  case,  as  the  prefer¬ 
ential  angle  of  polarization,  we  understand  the  angle 
corresponding  to  the  modal  value  of  the  empirical 
distribution  of  polarization  angles  over  the  pixels  cor¬ 
responding  to  the  background  scene.  We  initially 
focus  our  investigation  on  cases  1  and  2,  and  then 
compare  the  results  with  those  obtained  for  case  3. 
Practically,  the  APDI  system  images  the  scene  at  two 
consecutive  stages.  The  information  obtained  at  the 
first  stage  is  assigned  to  the  background  scene  (where 
no  target  is  present),  and  the  information  obtained 
at  the  second  stage  is  assigned  to  the  target  scene 
(where  the  target  and  the  background  are  present). 

4.  Validations  of  the  Adaptive  Polarization-Difference 
Imaging  Algorithm  and  the  Selection  of  the  Optimal 
Set  of  Parameters 

To  develop  the  APDI  approach  and  examine  its  util¬ 
ity,  we  conducted  several  sets  of  experiments  and 
simulations.  The  first  set  of  images  was  taken  in  our 
laboratory — a  controlled  environment  with  stable  il¬ 
lumination  conditions.  The  target  was  a  specially  de¬ 
signed  object  with  known  polarization  properties,  and 
the  background,  as  we  will  describe,  was  kept  simple, 
while  still  exhibiting  nonuniform  polarization  statis¬ 
tics.  This  enabled  an  accurate  evaluation  of  the  per¬ 
formance  of  APDI  for  the  target  detection.  The  second 
set  of  images  was  taken  with  real-life  targets  under 
natural  illumination  (sunlight). 

A.  Experimental  Setup  in  the  Laboratory 

The  laboratory  experimental  setup  and  a  specially 
manufactured  target  are  shown  in  Fig.  1.  An  in¬ 
candescent  150  W  lamp  illuminated  the  cylindrical 
Plexiglas  tank  (12  in.  in  height  and  16  in.  in  diame¬ 
ter)  from  a  side.  The  tank  was  filled  with  a  solution  of 
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Fig.  1.  (Color  online)  Layout  of 
the  experimental  setup:  (a)  pho¬ 
tograph  of  the  setup,  (b)  7-patch 
target  together  with  a  U.S.  dime 
(10-cent  coin). 


10  ml  of  whole  milk  diluted  in  approximately  27  1  of 
water.  The  height  of  the  water  level  in  the  tank  was 
21  cm.  This  created  a  model  of  dispersive  media  (such 
a  method  of  simulation  of  scattering  media  was  orig¬ 
inally  used  by  Tyo  et  al.12).  To  produce  Lambertian- 
type  illumination,  an  opal  glass  diffuser  was  placed 
between  the  tank  and  the  light  source.  The  target  was 
an  aluminum  disk  5.1  cm  in  diameter.  The  target 
surface  was  sandblasted,  and  there  were  seven  1  cm2 
square  patches  on  it  [see  Fig.  1(b)].  The  six  outer 
patches  were  abraded  with  emory  paper  in  such  a 
way  that  they  formed  three  orthogonally  oriented 
pairs,  i.e.,  0°  and  90°,  30°,  and  120°,  and  60°  and  150° 
(with  respect  to  the  vertical  axis).  Patches  with  or¬ 
thogonal  directions  in  the  scratches  are  located  dia¬ 
metrically  opposite  each  other.  The  surface  of  the 
center  patch  was  sandblasted  in  the  same  way  as  the 
base  plate  surface.  The  patches  were  raised  a  few 
mils  from  the  base  plate  surface.  The  target  was  at¬ 
tached  to  the  Plexiglas  plate  and  faced  up.  The  dis¬ 
tance  between  the  surface  of  the  water  and  the  plane 
of  the  aluminum  disk  was  55  mm.  The  target  was 
observed  with  an  Olympus  E-10  SLR  digital  camera 
with  a  Sunpack  62  mm  diameter  glass  polarizer  at¬ 
tached  in  front  of  it.  The  images  were  taken  sequen¬ 
tially  for  three  different  orientations  of  the  polarizer, 
i.e.,  0°,  45°,  and  90°  by  manually  rotating  the  polar¬ 
izer  between  shots.  All  the  images  in  the  laboratory 
experiment  were  taken  with  an  exposure  time  of  1  s 
and  the  f  number  of  the  camera  equal  to  4.0.  During 
such  exposure  time,  any  fluctuations  due  to  120  Hz 
oscillations  in  the  light  source  are  therefore  averaged 
out.  The  total  time  required  to  capture  all  three  im¬ 
ages  was  less  than  10  s  and  was  limited  mainly  by  the 
time  required  to  save  the  raw  image  of  the  scene  to 
the  camera's  flash  memory.  The  same  digital  camera 
and  polarizer  were  used  in  our  experiments  in  the 
natural  (uncontrolled)  environment. 

All  computer  analyses  in  this  study  have  been  done 
using  MATLAB  software  package  with  its  Image  Pro¬ 
cessing  Toolbox.  The  captured  images  were  stored  in 


an  Olympus  raw  format  (ORF),  which  gave  us  raw 
images  of  the  scene,  without  the  enhancements  and 
modifications  that  most  commercial  digital  cameras 
perform  internally  to  make  pictures  look  better.  The 
Olympus  E-10  has  a  single-chip  color  CCD  with  a 
RGGB  Bayer  primary  color  filter.  For  our  computa¬ 
tions,  we  extracted  the  R,  G,  and  B  components  of 
the  RGB  (red- green- blue)  output  directly  from  the 
RGGB  Bayer  filter  pattern  response,  so  that  the  im¬ 
age  had  only  of  the  total  number  of  active  pixels  of 
the  CCD  chip  (the  G  image  is  an  average  value  of  the 
two  G  filter  responses).  PCA  may  be  applied  to  any 
pair  of  polarization  channels  capturing  images  rep¬ 
resenting  one  of  the  components  (either  R,  G,  B,  or  V). 
The  V  (luminance)  component  of  the  hue-saturation- 
luminance  (HSV)  was  computed  by  the  MATLAB 
Image  Processing  Toolbox  and  was,  in  fact,  the  max¬ 
imum  value  of  the  R,  G,  or  B  channel  at  every  pixel. 
In  this  paper,  we  present  results  that  were  obtained 
based  on  the  V  component  of  the  images. 

The  camera  zoom  was  adjusted  in  such  a  way  that 
the  area  occupied  by  the  aluminum  disk  was  only  a 
portion  of  the  target  scene.  To  obtain  the  background 
scene,  we  simply  removed  the  aluminum  disk  from 
the  scene  while  keeping  all  other  experimental  con¬ 
ditions,  including  the  focusing  distance  of  the  camera, 
intact.  The  original  image  was  cropped  to  800  X 
600  pixels  for  efficiency  in  analysis.  The  polarization 
statistics  of  the  background  scene  are  shown  in  Fig.  2. 
The  histograms  of  the  polarization  parameters  reveal 
a  nonuniform  distribution  of  the  polarization  statis¬ 
tics  of  the  scene  with  the  average  degree  of  linear 
polarization  of  about  25%. 

Throughout  this  paper,  we  present  grayscale  im¬ 
ages  with  double  precision,  i.e.,  ranging  from  0  to  1, 
where  0  corresponds  to  0  and  1  corresponds  to  255  of 
8-bit  gray  scale.  To  assess  the  intrinsic  noise  of  the 
camera,  we  made  measurements  with  the  camera 
lens  covered  by  an  opaque  cap.  The  histograms  of 
the  standard  background's  half-intensity  distribution, 
i.e.,  Iu  are  compared  with  the  histogram  of  the  dark 
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Fig.  2.  (Color  online)  Normalized  histograms  of  polarization  parameters  of  the  background:  (a)  half  of  total  pixel  intensity  Iv\  (b)  degree 
of  linear  polarization  p;  (c)  angle  of  polarization  0.  Total  number  of  pixels  in  the  image  was  800  X  600.  Images  of  Iv  and  p  in  the  top  row 
are  stretched  to  cover  an  8-bit  gray-scale  range.  The  reason  we  have  systematic  variations  in  the  images  of  Iv  and  p  is  that  the  light  comes 
from  the  one  side  (top  left  corner). 


noise  image  in  Fig.  3.  The  mean  value  of  the  camera- 
noise  image  is  0.5  X  10“ 2  and  standard  deviation  is 
7  X  10“4,  while  for  the  background  intensity  image 
those  parameters  were  0.95  X  10-1  and  0.22  X  101, 
respectively.  Thus  intrinsic  camera  noise  is  negligible 
in  the  standard  conditions  of  the  experiments. 

B.  Application  of  the  Adaptive  Polarization-Difference 
Imaging  Algorithm  to  Images  Obtained  in  a 
Controlled  Environment 

Equation  (1)  was  applied  to  images  of  the  background 
taken  at  three  orientations  (<p  =  0°,  45°,  and  90°)  of 
the  linear  polarizer,  and  then  images  were  generated 
for  each  angular  orientation  of  the  polarizer  with¬ 
in  the  range  from  0°  to  180°  with  5°  steps  (cp  and 
cp  +  180°  are  indistinguishable  cases,  since  phase 


information  is  not  encoded  by  the  camera).  For  each 
pair  of  images  of  the  background  corresponding  to  the 
pair  of  analyzer  orientations  (cp1?  cp2),  we  then  ex¬ 
tracted  a  full  set  of  polarization  parameters  using 
PC  A,  i.e.,  a  =  a(cp1?  cp2),  P  =  P(<Pi,  92),  ki  =  M<Pi,  92), 
and  \2  =  X2(cpi,  cp2)  (Fig.  4). 

The  distributions  of  a(cp1?  cp2)  and  p(cpl7  cp2)  have 
similar  forms  and  in  particular  have  their  maxima 
and  minima  at  the  similar  locations  in  the  space 
(cpi,  <p2).  The  eigenvalues  and  \2  are  symmetric 
functions  of  (cp1?  cp2),  i.e.,  ^(cp1?  cp2)  =  \;(cp2,  <pi)  for 
i  =  1,2.  The  maximum  and  the  minimum  values  of  X1 
are  located  on  the  line  of  symmetry  (cp2  =  91),  which 
represents  the  situation  when  the  angle  of  the  linear 
polarizer  for  both  source  images  is  the  same  and  is 
obviously  not  useful.  Therefore  case  2  reduces  to  the 


Fig.  3. 


(a)  (b) 

Normalized  histogram  of  the  (a)  dark  noise  of  the  camera  compared  to  the  normalized  histogram  (b)  of  the  background  image. 
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(C)  (d) 

Fig.  4.  (Color  online)  Distribution  of  the  polarization  parameters  characterizing  the  standard  background  scene.  Each  panel  presents  a 
pseudocolor  representation  of  the  distribution  of  one  of  the  parameters:  (a)  a((p1?  <p2),  (b)  P(<pi,  <p2),  (c)  Xi(cpi,  92),  (d)  \2(<Pi>  92)-  The  scales  for 
each  parameter  are  provided  to  the  right  of  each  panel. 


analysis  of  behavior  of  the  smaller  eigenvalue,  i.e.,  X2. 
The  value  of  X2  represents  the  variance  in  the  PC2 
image,  suggesting  that  more  interesting  information 
can  be  obtained  from  the  PC2  image. 

The  APDI  algorithm  was  applied  to  each  target- 
background  pair,  and  PC1  and  PC2  images  for  all 
three  cases  described  in  Subsection  3.C  were  com¬ 
puted,  and  the  corresponding  parameters  are  given  in 
Table  1.  PC1  and  PC2  images  for  all  three  cases  are 
illustrated  in  Fig.  5.  The  PC1  images  for  the  three 
cases  are  very  similar,  and  in  particular,  the  nonuni¬ 
form  illumination  of  the  scene  is  clearly  visible  in 
each.  The  PC2  images  in  panels  (b)  and  (c)  of  Fig.  5  are 
similar  because  the  pairs  of  angles  derived  for  cases 
2  and  3  are  close  to  each  other,  as  well  as  the  corre¬ 
sponding  polarization  parameters  (see  Table  1).  How¬ 
ever,  the  PC2  image  corresponding  to  case  1  [Fig.  5(a)] 


Table  1.  Adaptive  Parameters  Corresponding  to  the  Cases  Considered 
for  the  Benchmark  Target 


Case 

<Pi° 

92° 

a 

p 

X  1(T3 

\2  X  1(T5 

1 

145 

55 

0.895 

-0.446 

1.116 

0.611 

2 

95 

5 

0.680 

-0.733 

1.005 

3.447 

3 

100 

10 

0.707 

-0.707 

1.004 

3.327 

is  noticeably  different:  The  margins  of  the  disk  are 
more  clearly  outlined,  and  the  direction  of  scratches 
in  all  the  patches  (which  vary  in  polarization  orien¬ 
tation)  are  distinguishable.  The  clear  segregation  of 
the  disk  from  the  background  in  the  PC2  image  is 
achieved  in  a  large  part  due  to  the  lower  variance  of 
the  background  scene  in  this  channel,  as  we  now 
describe. 

In  Fig.  6,  we  present  normalized  (i.e.,  unit  area) 
histograms  of  the  PC  images  for  all  cases  considered. 
The  distributions  and,  in  particular,  the  standard 
deviations  of  the  PC1  images  are  very  close  for  all 
three  cases  (o^2  ~  0.5  X  10-2).  In  contrast,  for 
the  PC2  images  the  standard  deviation  for  case  1 
(ax2  =  3.68  X  10-3)  is  more  than  50%  lower  than  in 
the  other  two  cases  (case  2,  a22  =  8.62  X  10“3;  case  3, 
o-32  =  8.53  X  10~3).  Thus  the  PC2  image  for  case  1  has 
the  smallest  variance  in  the  distribution  of  pixel  in¬ 
tensities,  suggesting  that  case  1  may  provide  an  “op¬ 
timum”  set  of  adaptive  parameters  for  the  detection 
of  novel  targets  in  PC2  images. 

C.  Performance  Evaluation  of  Adaptive 
Polarization-Difference  Imaging:  Sensitivity  Index 

The  selection  of  the  “optimum”  transformation  ma¬ 
trix  for  a  specific  background  should  both  lead  to  a 
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PC,  PC 2 


(c) 


Fig.  5.  Principal  components  of  the  scene  corresponding  to  three  cases  of  interest.  Left  column  shows  PCi,  and  right  column  PC2  images, 
respectively.  Panels  (a)-(c)  correspond  to  cases  1-3,  respectively.  All  the  images  are  linearly  rescaled  to  exploit  the  8-bit  displayable  range. 
The  size  of  the  images  was  800  X  600  pixels. 


minimum  variance  in  the  PC2  image  of  the  back¬ 
ground,  and  enable  targets  with  novel  polarization 
properties  to  pop  out  in  the  PC2  channel.  To  avoid 
relying  only  on  the  evaluation  of  the  images  of  the 
targets  by  human  viewing,  we  employed  an  objective 
numerical  evaluation:  the  sensitivity  index  calcula¬ 
tion.  The  sensitivity  index  is  based  on  the  signal- 
detection  theory  (SDT),  which  quantifies  an  observer's 
ability  to  discriminate  a  target  from  a  background.56  In 


SDT,  an  observation  taken  at  some  moment  may  arise 
from  a  noise-alone  distribution,  or  from  the  signal- 
plus-noise  distribution,  with  the  means  of  these  distri¬ 
butions  separated  by  a  certain  amount  da  specified  in 
units  of  the  standard  deviations;  da  is  called  the  sen¬ 
sitivity  index.  With  an  increase  of  da,  the  probability 
of  successful  target  detection  (a  hit)  will  increase, 
since  the  overlap  between  the  distributions  de¬ 
creases,  and  the  probability  of  a  false  alarm  will  also 
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Fig.  6.  Normalized  histograms  of  (a)  PC^  and  (b)  PC2  for  all  the  cases  shown  as  images  in  Fig.  5.  The  standard  deviations  of  PCs  for  all 
three  cases  considered  are  shown  in  the  figures. 


decrease.  In  our  study  as  the  signal-plus-noise  distri¬ 
bution,  we  consider  the  scene  distribution  including 
both  the  target  and  the  background,  while  the  noise- 
alone  distribution  is  taken  as  the  scene  distribution 
without  the  target  present. 

To  estimate  da  empirically,  we  performed  a  series 
of  measurements  with  3  min  separation  between 
each  set.  The  experimental  setup  was  as  described  in 
Subsection  4.A.  We  first  captured  20  sets  of  images  of 
the  background  alone,  and  then  20  sets  of  images  of  the 
background-plus-target  scenes;  each  set  of  images 
is  composed  of  three  images  collected  at  9  =  0°, 
45°,  and  90°,  allowing  the  derivation  of  an  equivalent 
image  for  any  polarizer  orientation  cp  and  PCA,  yield¬ 
ing  PC1  and  PC 2  images  for  cases  1-3.  An  observation 
for  the  SDT  analysis  was  computed  over  two  square 
regions  of  5  X  5  pixels,  determined  by  the  imaged 
location  of  special  portions  of  the  target  (Fig.  7).  For 
each  of  the  20  sets  of  images  of  the  background  and 
the  background-plus-target  scenes,  the  average  in¬ 
tensities  of  these  regions  in  the  PC1  and  PC2  images 
were  calculated.  In  summary,  we  ended  up  with  20 


Fig.  7.  Scheme  of  the  specially  created  target  where  regions  used 
for  the  sensitivity  index  calculation  are  marked  in  black.  Dashed 
lines  identify  the  direction  of  scratches  in  the  specific  patch.  The 
left  region  is  referred  to  as  region  one  and  the  right  region  is 
referred  to  as  region  two,  respectively. 


pairs  of  observations  of  the  PCX  and  PC2  values 
for  these  two  specific  regions  of  the  scene  for  each  of 
the  three  cases.  We  then  estimated  the  sensitivity 
index  as 

I*- to  (7) 

<WW)/2 

where  jjl5  and  crB 2  are  the  mean  and  the  variance  of 
the  background  scene  in  the  specific  region,  and  jjlt 
and  crT2  are  the  mean  and  the  variance  of  the  same 
region  when  the  target  object  is  present.56  The  values 
of  the  sensitivity  index  for  PC2  images  corresponding 
to  the  three  cases  (and  the  two  target  regions,  res¬ 
pectively)  were  as  follows:  case  1,  da  =  1.703  and 
da  =  1.687;  case  2,  da  =  0.772  and  da  =  1.371;  case  3, 
da  =  1.419  and  da  =  0.908,  respectively.  Here,  the 
first  value  of  da  in  each  case  corresponds  to  the  left- 
hand  side  target  region,  and  the  second  value  corre¬ 
sponds  to  the  right-hand  side  region,  (Fig.  7).  The 
parameters  determined  by  case  1  clearly  yield  a  su¬ 
perior  detectability  of  the  polarization  targets  than 
those  determined  by  cases  2  and  3.  Remarkably,  the 
case  1  PC2  channel  outperforms  the  PC2  channel  of 
the  other  two  cases  on  both  the  left  and  the  right 
targets,  even  though  each  of  these  other  channels 
performs  much  better  on  one  of  the  two  targets.  These 
observations  suggest  that  the  PC2  channel  generated 
with  the  maximum  components  of  eigenvectors  (a,  (3) 
correspond  to  the  “optimum”  pair  of  angles  for  the 
detection  of  the  polarization  targets  against  a  polar¬ 
ized  background. 

5.  Target  Detection  Against  a  Nonuniformly  Polarized 
Background  Under  Natural  Illumination  Conditions 

In  addition  to  the  experiments  in  the  laboratory,  we 
performed  several  experiments  under  natural  illumi¬ 
nation  conditions.  As  in  the  laboratory,  every  exper¬ 
iment  session  includes  capturing  three  images  of  both 
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the  background  scene  and  the  target  scene.  Unlike 
the  laboratory  experiments,  where  the  background 
was  a  largely  a  uniform  scene,  in  this  situation  the 
background  is  a  relatively  complex  one,  and  contains 
several  different  objects. 

The  first  experiment  presented  here  was  done  in¬ 
side  the  Levine  Hall  of  the  University  of  Pennsylva¬ 
nia.  The  images  for  this  experiment  show  the  floor 
surface  of  the  hall  (Fig.  8).  The  scene  consisted  of  the 
shadow  region  formed  on  the  floor  by  the  combined 
shadows  of  a  person  standing  upright  and  the  win¬ 
dow  frame.  The  rest  of  the  scene  was  illuminated  by 
the  sunlight  passing  through  the  window  glass.  The 
camera  was  located  opposite  the  window  and  ele¬ 
vated  above  the  floor;  thus  the  scene  was  illuminated 
by  sunlight  in  such  a  way  that  the  light  beam  coming 
from  the  Sun  and  a  beam  reflected  from  the  scene 
formed  the  same  plane.  The  floor  was  a  dark  gray 
color  with  a  broad  pattern.  The  target  object  was  the 
translucent  plastic  case  of  a  CD.  We  took  three  im¬ 
ages  of  the  background  and  of  the  target  scenes.  The 
camera  settings  were  kept  the  same  for  both  the 
background-only  image  sets  and  the  image  sets  with 
the  target  in  the  scene.  The  exposure  time  for  each 
image  was  1/8  and  the  /‘number  of  the  camera  was 
equal  to  8.  Figure  8  shows  three  polarization  compo¬ 
nents  (  Ijj ,  p,  and  0)  for  both  the  background  and  the 
target  scenes,  placed  side  by  side  for  easy  comparison: 
The  target  object  has  a  lower  degree  of  polarization 
than  the  surrounding  background. 

Using  the  ADPI  case  1  algorithm,  the  “optimum” 
pair  of  angles  are  found  to  be  cpx  =  140°  and  cp2  =  50°, 
with  eigenvector  components  a  =  0.9965  and 
(3  =  -0.0834.  The  histogram  of  the  background  angle 
of  polarization  shows  that  the  preferential  angle  of 
polarization  of  the  background  is  approximately 
G#  =  140°,  which  means  that  the  optimal  angle  pair  is 
the  angle  of  preferential  polarization  and  the  angle 
orthogonal  to  it.  The  angles  corresponding  to  the 
maximum  and  the  minimum  of  the  larger  eigenvalue 
(i.e.,  are  found  in  the  bisector  line  of  the  eigen¬ 
value  surface,  where  \!max  is  located  at  the  point 
9i  =  cp2  =  0s?  and  \rm  is  located  at  the  point 
cpi  =  cp2  =  0s  —  90°.  This  information  can  be  useful:  If 
the  preferential  polarization  angle  is  known  a  priori , 
then  the  optimal  angle  pairs  can  be  found  directly 
without  time-consuming  computation.  On  the  other 
hand,  this  can  be  an  effective  way  to  recover  the 
preferred  polarization  direction  in  a  scene. 

Comparing  the  principal  component  images  of  the 
scene  [panels  (b)  and  (d)  in  Fig.  9],  one  may  see  a 
significant  improvement  in  the  target- background 
contrast  of  the  PC2  image  over  the  conventional  PD 
image.  To  have  a  more  comprehensive  comparison, 
we  also  included  images  that  correspond  to  the  pair  of 
angles  bringing  the  smaller  eigenvalue  to  the  maxi¬ 
mum  (case  2);  i.e.,  consider  images  with  the  minimum 
variance  in  intensity.  For  this  experiment,  the  maxi¬ 
mum  value  of  X2  is  found  at  cpx  =  105°,  cp2  =  15°, 
and  the  corresponding  coefficients  are  a  =  0.5016, 
P  =  -0.8650.  Principal  component  images  created  in 


this  case  are  also  presented  in  Fig.  9  [panels  (g)  and 
(h)] .  Although  an  improvement  over  the  conventional 
PD  image  (case  3)  is  noticeable  [comparing  panels  (d) 
and  (h)  in  Fig.  9],  the  pair  of  angles  determined  by 
case  1  yields  better  results.  Overall,  then,  we  con¬ 
clude  that  case  1  yields  the  “optimum”  PC2  channel. 

6.  Discussion 

We  have  presented  the  mathematical  basis  of  a  novel 
adaptive  PDI  technique,  and  experimental  results 
that  demonstrate  its  performance  in  target  detection 
applications.  For  scenes  with  uniform  distributions  of 
polarization  parameters,  APDI  reduces  to  a  conven¬ 
tional  PDI  technique.  However,  in  many  real-world 
situations,  APDI  will  yield  superior  performance,  so 
that  targets  with  polarization  features  can  be  more 
readily  detected  against  a  background. 

A.  Summary  of  the  Adaptive  Polarization-Difference 
Imaging  Algorithm 

For  the  discussion  to  follow,  it  is  useful  to  summarize 
the  APDI  algorithm: 

1.  Capture  three  images  of  the  background  scene 
for  three  different  orientations  9  of  the  linear  polar¬ 
ization  analyzer. 

2.  Process  those  three  images  and  obtain  complete 
polarization  statistics  of  the  background  scene  ac¬ 
cording  to  Eq.  (2). 

3.  Synthesize  polarization  images  of  the  scene  for 
the  full  range  of  angles  of  orientation  cp  of  the  polar¬ 
ization  analyzer  using  Eq.  (1). 

4.  Perform  PCA  on  all  possible  pairs  of  angles  and 
obtain  four  adaptive  parameters  as  the  functions  of 
angle  of  the  polarizer’s  orientation,  the  two  eigenvec¬ 
tor  components,  and  the  two  eigenvalues. 

5.  Find  the  “optimum”  set  of  adaptive  parameters 
(case  1)  and  create  the  transformation  matrix  of 
the  background  scene,  i.e.,  TBG  (<Pi°P\  920pt). 

6.  Repeat  steps  1  and  2  for  the  scene  with  the 
target  present. 

7.  Synthesize  a  pair  of  polarization  images  of  the 
target  scene  for  optimal  pair  of  angles  (cpi°pt,  cp2opt)  as 
found  in  step  5. 

8.  Using  the  transformation  matrix  that  was  ob¬ 
tained  from  the  background  scene,  TBG  (<Pl°Pt,  <P2°Pt), 
create  principal  component  images  of  the  target  scene 
according  to  Eq.  (6). 

Since  the  APDI  algorithm  deals  with  the  polariza¬ 
tion  statistics  of  the  scene,  if  the  target  occupies  only  a 
small  portion  of  the  scene,  adaptive  coefficients  and 
optimal  pairs  of  angles  obtained  from  statistics  of 
the  background  scene  will  differ  little  from  those  ob¬ 
tained  from  the  statistics  of  the  target  scene.  Thus  the 
APDI  algorithm  may  be  applied  to  the  target  scene 
directly  without  gathering  additional  information  from 
the  background.  However,  if  the  target  object  occupies 
a  significant  portion  of  the  image,  a  complete  set  of 
measurements  of  the  background  is  required  to  segre¬ 
gate  the  target  object  from  the  background.  Although 
in  this  study  we  dealt  with  scenes  that  had  a  nonuni- 
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Fig.  8.  Polarization  components  for  the  nontarget  and  target  scenes  of  the  experiments  under  natural  lighting.  The  left  panel  shows  Iv, 
p,  and  0  (top  to  bottom)  images  of  the  nontarget  scene,  and  the  right  panel  shows  those  for  the  target  scene.  The  Iv  plots  in  both  cases  are 
linearly  rescaled  to  use  the  full  8-bit  gray-scale  display  range. 
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Fig.  9.  Comparison  in  target  detection  between  images  obtained  by  our  new  adaptive  algorithm  and  by  the  conventional  PDI  algorithm. 
They  are  the  principal  component  images  obtained  from  the  images  shown  in  Fig.  8.  Panels  (a)  and  (b)  are  PCi  and  PC2  for  case  1.  Panels 
(c)  and  (d)  are  conventional  PS  and  PD  images  (case  3).  Panels  (e)  and  (f)  are  PCX  and  PC2  for  case  2.  All  images  were  linearly  rescaled 
to  cover  an  8-bit  gray-level  display  range. 
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(a)  (b) 

Fig.  10.  (a)  SNR  for  PCi  and  PC2.  Here,  the  signal  is  the  area  of  the  aluminum  disk,  and  the  noise  is  the  rest  area  of  the  corresponding 

PC.  (b)  Normalized  histograms  of  the  PC2  images  with  a  shift  from  the  optimal  (case  1)  pair  of  angles.  Increasing  variance  in  the  PC2  image 
with  rotation  of  the  optimal  pair  of  angles  is  shown. 


form  distribution  in  polarization  parameters,  the  po¬ 
larization  of  the  background  had  only  one  preferential 
direction  of  polarization.  If  the  scene  has  two  or  more 
regions  of  significant  areas  that  have  different  angles 
of  preferential  polarization,  further  improvements 
may  be  required,  such  as  segmentation  of  the  scene 
based  on  its  polarization  statistics.  Such  results  will  be 
reported  in  a  subsequent  paper. 

The  APDI  algorithm  is  relatively  fast  and  does  not 
require  significant  computer  resources.  For  the  com¬ 
puter  workstation  with  a  Pentium  4  550  processor 
and  2  GB  of  RAM,  the  total  computational  time  was 
about  30  s  using  the  code  written  in  MATLAB.  Re¬ 
writing  the  code  using  a  less  resource  aggressive  pro¬ 
gramming  language,  such  as  Gf+  will  significantly 
reduce  the  processing  time.  The  time  required  for 
capturing  the  images  may  also  be  reduced.  Currently, 
in  the  laboratory,  we  employ  a  setup  where  the  rota¬ 
tion  of  the  polarizer  is  performed  by  a  stepping  motor 
and  the  entire  process  can  be  controlled  from  the 
computer.  Use  of  this  setup  will  automate  the  proce¬ 
dure  of  capturing  images  and  their  transition  to  the 
computer. 

B.  Potential  Applications 

Based  on  the  computer  simulations  and  experiments 
presented  above,  we  propose  the  APDI  algorithm  for 
use  in  the  detection  of  targets  with  polarization  fea¬ 
tures.  To  speed  the  process,  the  initial  pair  of  angles 
may  be  selected  such  that  one  of  the  angles  is  equal  to 
the  preferential  polarization  angle  of  the  background 
and  the  other  angle  is  chosen  to  be  orthogonal  to  the 
preferential  angle.  To  ensure  that  the  selected  pair  is 
“optimum,”  the  calculations  of  adaptive  coefficients 
for  a  few  pairs  of  polarizer  orientations  around  the 
selected  point  should  be  performed.  The  pair  with  the 
maximum  component  value  of  the  eigenvectors  is 
then  used  as  the  optimal  pair.  This  approach  may 


significantly  decrease  the  computational  time,  as  is 
very  important  for  real-time  applications. 

C.  Sensitivity  of  the  Adaptive  Polarization-Difference 
Imaging  Algorithm  to  the  Rotation  of  the  Pair  of  Angles 

Since  the  APDI  algorithm  may  be  used  as  an  effective 
tool  in  a  visual  surveillance  system,57  an  important 
issue  in  this  application  is:  “How  sensitive  are  the 
adaptive  pairs  to  the  rotation  of  the  polarization 
channels?” 

Assume  that  by  applying  the  APDI  algorithm,  an 
“optimum”  set  of  (case  1)  parameters  has  been  found. 
Consider,  then,  rotating  this  pair  by  a  certain  angle 
clockwise  or  counterclockwise.  Would  it  be  possible  to 
obtain  a  PC2  image  of  the  same  quality?  A  simulation 
of  the  sensitivity  of  the  APDI  algorithm  to  such  rota¬ 
tions  was  undertaken  with  the  same  set  of  experi¬ 
mental  data  reported  above.  The  adaptive  coefficients 
were  applied  to  the  pair  of  images  corresponding  to 
angles  rotated  by  5°  and  10°  from  the  “optimum” 
(case  1)  pair.  The  goal  was  to  check  in  which  case  the 
aluminum  disk  is  better  able  to  be  detected  against 
the  surrounding  water  solution.  For  each  case,  we 
calculated  the  SNR,  considering  the  output  from  the 
aluminum  disk  as  the  signal  and  the  output  from  the 
surrounding  water  solution  as  the  noise  or  back¬ 
ground.  Thus  we  derived 


where  p,#  is  the  mean  value  of  pixels  belonging  to  the 
aluminum  disk,  jjlw,  o>  are  the  mean  value  and  the 
standard  deviation  of  all  the  images  except  the  alumi¬ 
num  disk,  respectively.  Figure  10(a)  shows  SNR  for 
PC1  and  PC2  corresponding  to  optimal  pair  angles  and 
for  pairs  of  angles  rotated  by  5°  and  10°  with  respect  to 
the  optimal  pair.  We  note  that  the  SNR  for  the  optimal 
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(C)  (d) 

Fig.  11.  SNRs  for  the  PC1?  PC2,  PS,  PD,  and  p  with  the  presence  of  (a)  artificially  added  Gaussian  noise,  and  (b)  white  noise.  Images  of 
(c)  PC2  and  (d)  p  for  5%  of  added  Gaussian  noise.  Individual  features  of  the  aluminum  disk,  such  as  the  appearance  of  patches,  is  better 
visible  in  the  PC2  image. 


pair  of  angles  is  the  highest  and  rapidly  decreases  with 
a  rotating  of  the  polarizers.  With  rotation  of  the  angle 
pair,  both  the  background  level  and  its  variance  in¬ 
crease,  while  the  intensity  level  of  the  target  remains 
at  approximately  the  same  level.  This  can  be  seen  in 
the  histograms  of  the  PC2  images  for  the  area  sur¬ 
rounding  the  aluminum  disk  [see  Fig.  10(b)].  These 
results  show  that  a  shift  in  the  pair  of  angles  from 
those  determined  by  case  1  may  decrease  the  SNR  for 
a  target  object  against  the  surrounding  background. 

D.  Effects  of  Noise  on  the  Performance  of  the  Adaptive 
Polarization-Difference  Imaging 

In  this  subsection,  we  discuss  the  influence  of  noise  on 
the  performance  of  our  technique.  To  evaluate  the  per¬ 
formance  of  APDI,  we  artificially  added  noise  to  a  set  of 
input  images,  I0,  /45,  and  I90  for  both  background  and 
target  scenes.  We  used  two  types  of  noises,  i.e.,  white 
noise  and  Gaussian  noise  with  mean  values  in  both 
cases  equal  to  the  mean  values  of  the  input  images. 
The  standard  deviation  for  the  Gaussian  noise  was 
selected  as  a  certain  percentage  of  the  dynamic  range 
of  the  input  images;  i.e.,  the  mean  value  of  the  input 
images  was  multiplied  by  a  coefficient  that  varied  from 
0.01  to  0.21.  For  the  white  noise,  the  range  of  varia¬ 
tions  was  the  same  as  the  standard  deviation  in  the 


case  of  the  Gaussian  noise.  Once  the  noise  was  added 
to  the  input  images,  the  polarization  components  were 
recovered,  and  then  the  APDI  procedure  as  described 
in  Subsection  6.A  was  done.  For  each  percentage  of 
noise,  we  calculated  SNR  according  to  Eq.  (8)  for  both 
PCs  corresponding  to  optimal  pair  of  angles,  for  con¬ 
ventional  PS  and  PD  images,  and  for  the  degree  of 
linear  polarization  p.  The  results  are  shown  in  Fig.  11. 
We  may  note  that  APDI  is  more  sensitive  to  the  pres¬ 
ence  of  the  Gaussian  noise  than  to  the  white  noise. 
After  a  certain  amount  of  noise,  the  SNR  calculated 
using  the  image  of  p  becomes  slightly  higher  than 
those  for  the  PC2,  although  individual  features  of  the 
target,  such  as  patches  are  not  detected  in  the  image  of 
p;  it  is  detected  in  PC2  [compare  panels  (c)  and  (d)  in 
Fig.  11,  where  the  images  are  shown  with  an  addition 
of  5%  of  noise] .  With  an  increasing  amount  of  noise,  the 
performance  of  the  APDI  algorithm  decreases,  but  still 
for  the  case  of  white  noise  even  with  the  amount  of 
noise  about  20%,  the  performance  of  APDI  is  higher 
than  those  of  the  conventional  PDI. 

E.  Adaptive  Polarization-Difference  Imaging  Compared  to 
Other  Polarization  Analysis  Techniques 

To  produce  the  PD  images,  either  a  scene  is  illumi¬ 
nated  by  natural  light  and  the  scattered  light  is  an- 
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(a) 


Fig.  12.  Stretched  images  of  (a) 
PC2  and  (b)  p.  (c)  Normalized 
histograms  of  images  shown  in 
panels  (a)  and  (b).  Standard 
deviations  for  PC2  and  p  are 
cr2  =  0.0054  and  a2  =  0.041,  re¬ 
spectively. 


alyzed  with  the  two  orthogonal  polarizers,  or  a  scene 
is  illuminated  with  the  two  light  sources  of  orthogo¬ 
nal  polarizations.  The  idea  of  weighted  subtraction  of 
the  two  orthogonal  components  of  the  scene  was  first 
introduced  by  Walker  et  al.18  Their  method  involves 
subtraction  of  a  scaled  image  obtained  at  one  polar¬ 
ization  from  the  oppositely  polarized  image,  and 
shows  improvement  over  methods  in  which  subtrac¬ 
tion  is  done  without  scaling.  In  this  study,  the  image 
contrast  was  evaluated  as  a  function  of  the  scaling  of 
the  subtracted  image,  and  the  orthogonal  polariza¬ 
tion  axes  were  fixed.  In  contrast,  the  APDI  algorithm 
presented  here  adaptively  selects  two  orientations  of 
the  polarizer,  and  also  finds  the  “optimum”  weighting 
of  the  resultant  images  by  employing  PCA  to  derive 
the  transformation  matrix  [Eqs.  (5)  and  (6)],  and 
thereby  the  resultant  PC1  and  PC2  images. 

Several  publications  have  discussed  the  nonlinear 
aspects  of  polarization  imaging  techniques  (see,  e.g., 
Refs.  10,  11,  and  13).  The  APDI  algorithm  proposed 
here  is  essentially  linear  once  PCA  analysis  has  been 
applied  to  the  background  [derivation  of  the  images  of 
the  background  corresponding  to  a  full  set  of  orien¬ 
tations  of  the  polarizer  involves  the  nonlinear  equa¬ 
tions,  Eq.  (1)  and  (2)].  An  advantage  of  the  APDI 
method  is  that  it  is  readily  applicable  to  many  differ¬ 
ent  kinds  of  scenes  in  which  polarization  affects  im¬ 
age  intensities,  including  scenes  that  include 
specular  reflection  (e.g.,  Fig.  9),  underwater  scenes 
(Fig.  5),  and  low-light  scenes  in  which  either  natural 
or  artificial  illumination  is  used.  Moreover,  APDI  can 
be  implemented  without  using  specialized  equip¬ 
ment:  only  a  digital  camera  and  a  polarizer  are  re¬ 
quired  for  capturing  images. 

It  has  been  proposed  that  images  of  the  degree  of 


linear  polarization  in  a  scene6’7’19  [cf .p(x,  y)  in  Eq.  (2)] 
may  provide  a  valuable  tool  for  the  detection  and 
discrimination  of  objects.  Such  images,  no  doubt, 
have  considerable  value,  and  can  be  readily  imple¬ 
mented  with  the  tools  employed  here.  While  we  have 
not  performed  an  exhaustive  comparison,  we  can  con¬ 
fidently  assert  that  the  signal  obtained  with  PC2  of 
APDI  outperforms  that  obtained  from  p  images  in  the 
cases  we  have  examined;  one  such  comparison  is  pro¬ 
vided  in  Fig.  12.  In  this  case,  the  variance  of  the 
background  in  the  PC2  image  is  about  1/10  that  of  the 
p  image.  As  a  result,  the  SNR  calculated  according  to 
Eq.  (8)  is  higher  for  the  PC2  versus  the p  image:  3.056 
versus  2.417.  Nonetheless,  for  the  superior  SNR  of 
APDI,  under  conditions  where  speed  is  critical,  use  of 
the  p  image  for  rapid  initial  inspection  should  be 
quite  valuable,  and  it  bears  emphasis  that  the  p  im¬ 
age  data  are  automatically  generated  in  the  APDI 
algorithm. 

7.  Conclusions 

We  have  developed  a  set  of  techniques  to  form  an 
“optimum”  linear  combination  for  the  polarization 
channels  that  is  adapted  to  the  polarization  statistics 
of  a  scene.  Utilizing  the  technique  of  PCA,  we  have 
determined  an  optimum  linear  combination  of  polar¬ 
ization  channels  [Eqs.  (5)  and  (6)]  to  produce  PC2 
images  that  efficiently  provide  information  for  dis¬ 
criminating  a  target  with  polarization  properties 
from  the  background  scene.  The  adaptive  transfor¬ 
mation  is  readily  adjusted  as  the  imaging  system 
observes  different  environments  or  varying  lighting 
conditions.  The  adaptive  transformation  is  particu¬ 
larly  suitable  for  environments  with  preferential  po¬ 
larization  distribution.  This  approach  may  point  to 


5518  APPLIED  OPTICS  /  Vol.  45,  No.  22  /  1  August  2006 


an  interesting  research  direction  in  the  polarization 
vision  in  certain  aquatic  species,  which  may  have 
detector  arrays  that  act  like  an  adaptive  PC2  channel. 
A  further  utility  of  the  APDI  system  is  that  once  the 
polarization  information  [see  Eq.  (2)]  fully  character¬ 
izing  the  background  scene  has  been  collected  (e.g., 
by  a  surveillance  system  that  routinely  takes  images 
at  three  orientations  of  a  linear  polarizer),  the  image 
data  may  be  processed  off-line  to  yield  an  optimum 
presentation  of  the  polarization  features  of  the  scene 
that  may  otherwise  escape  attention. 
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Abstract:  Shadow  is  an  inseparable  aspect  of  all  natural  scenes.  When  there 
are  multiple  light  sources  or  multiple  reflections  several  different  shadows 
may  overlap  at  the  same  location  and  create  complicated  patterns.  Shadows 
are  a  potentially  good  source  of  information  about  a  scene  if  the  shadow 
regions  can  be  properly  identified  and  segmented.  However,  shadow  region 
identification  and  segmentation  is  a  difficult  task  and  improperly  identified 
shadows  often  interfere  with  machine  vision  tasks  like  object  recognition 
and  tracking.  We  propose  here  a  new  shadow  separation  and  contrast 
enhancement  method  based  on  the  polarization  of  light.  Polarization 
information  of  the  scene  captured  by  our  polarization-sensitive  camera  is 
shown  to  separate  shadows  from  different  light  sources  effectively.  Such 
shadow  separation  is  almost  impossible  to  realize  with  conventional, 
polarization-insensitive  imaging. 
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Polarization;  (230.5440)  Polarization- sensitive  devices 
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1.  Introduction 

Shadows  are  formed  whenever  an  occlusion  partially  blocks  the  illumination  of  a  surface  or 
object  by  a  light  source.  With  the  exception  of  the  ambient  light,  which  is  assumed  to  be 
omnidirectional,  light  sources  illuminate  surfaces  from  only  one  specific  direction.  In  addition 
to  classification  by  the  source  direction,  shadows  are  further  classified  into  “self’  and  “cast”. 
A  “self’  shadow  refers  to  the  regions  of  an  object  not  directly  illuminated  by  a  light  source 
due  to  its  surface  orientation,  whereas  “cast”  shadow  refers  to  a  region  not  illuminated  by  a 
source  due  to  occlusion  by  other  objects.  Shadowed  regions  usually  appear  darker  than  the  lit 
regions  and  their  color  properties  (e.g.,  hue  and  saturation)  can  also  appear  different  than  the 
directly  illuminated  regions.  Such  differences  in  intensity  and  color  create  patterns  and 
boundaries/edges  that  often  confuse  human  observers  or  machine  vision  algorithms  which 
attempt  to  segment  scenes  and  identify  objects  using  these  cues.  For  this  reason  many 
techniques  have  been  developed  to  identify,  segment,  and  remove  shadows  from  an  image  or 
a  video  sequence[l-19].  However,  all  previously  published  methods  use  only  two  aspects  of 
light,  its  intensity  and/or  spectral  (“color”)  distribution  as  information  in  shadow 
segmentation,  though;  in  some  cases  these  are  combined  with  temporal  and  geometric 
information  available.  It  appears  that  a  third  fundamental  property  of  light,  its  polarization, 
has  not  heretofore  been  used  for  the  purpose  of  shadow  segmentation.  Furthermore,  most 
existing  shadow  segmentation  algorithms  assume  a  relatively  simple  shadow  model:  an  area 
of  a  scene  is  classified  either  as  shadow  or  non-shadow.  In  fact  it  is  possible  for  a  specific 
region  of  a  scene  to  be  both  shadow  for  one  source  and  illuminated  simultaneously  by  another 
source  or  sources,  as  explained  below,  and  polarization  information  can  assist  in  “parsing” 
such  complications  in  scene  segmentation. 

Polarization  is  an  intrinsic  property  of  light.  Light  from  the  dominant  natural  source,  the 
sun,  is  not  polarized  but  light  scattered  from  small  particles  in  the  sky  and  most  light  reflected 
or  scattered  from  object  surfaces  is  partially  polarized.  The  unaided  human  eye  and  most 
machine  vision  cameras  are  “blind”  to  polarization,  but  some  animal  species  can  detect  and 
utilize  polarization  information  and  use  it  for  a  variety  of  purposes,  including  navigation  and 
object  recognition  [20-23].  Inspired  by  biological  polarization  vision,  our  group  has  developed 
polarization  sensitive  cameras  and  processing  methods  for  the  detection  of  targets  in 
scattering  media,  detecting  latent  fingerprints  and  enhancing  surveillance  [24-28].  We  have 
also  developed  methods  for  displaying  polarization  information  effectively  to  human 
observers  [29,30].  It  has  been  reported  that  polarization  increase  in  dark  surface  area  [31],  and 
that  polarization  can  be  used  to  enhance  details  in  shadow  [32].  It  has  also  been  reported  that 
polarization  increases  with  increasing  incident  light  angle  [33].  In  this  investigation  we  show 
that  complex  overlapping  cast  shadows  that  are  almost  impossible  to  distinguish  in  images 
generated  with  only  intensity  and  color  information  can  be  readily  segmented  from  each  other 
in  images  generated  from  the  polarization  parameters  of  a  scene. 
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2.  Polarization  and  Shadow 


Unpolarized 
incident  light 


Specular  reflection 
Diffuse  reflection 


(a) 


(b) 


Fig.  1.  (a)  General  macroscopic  reflection  model  (b)  Polarization  of  light  resulting  from 
specular  reflection  from  a  dielectric  surface. 

According  to  the  generally  accepted  macroscopic  description  of  the  interaction  of  light  with  a 
surface,  reflected  light  can  be  subdivided  into  specular  and  diffuse  components  [Fig.  1(a)]. 
The  ratio  of  energy  carried  by  the  diffuse  and  specular  components  depends  on  the  angle  of 
incidence  and  the  material  properties  of  the  surface.  The  diffusely  reflected  components  often 
undergo  multiple  random  reflections  microscopically,  so  statistically  they  tend  to  be 
unpolarized.  In  contrast,  the  specularly  reflected  component  is  usually  at  least  partially 
polarized,  with  the  polarization  direction  (dominant  axis  of  E-field  oscillation)  parallel  to  the 
local  tangent  plane  of  the  surface  [Fig.  1(b)].  These  physical  phenomena  can  be  formalized 
through  appropriate  application  of  Fresnel’s  analysis  and  Eq.  [34]. 

In  addition  to  the  scattering  by  surfaces,  another  important  natural  source  of  polarization  is 
the  scattering  of  light  by  the  atmosphere  of  the  earth.  The  polarization  of  sun  light  by  the  air 
particles  can  be  explained  by  the  theory  of  Rayleigh  scattering  [34],  which  describes  the 
particles  as  electric  dipoles:  because  oscillating  dipoles  do  not  radiate  in  the  direction  of 
oscillation,  a  polarization-sensitive  observer  will  see  the  dome  of  the  sky  to  exhibits 
polarization  pattern  that  depends  on  the  location  of  the  sun.  Since  pioneering  investigations  of 
von  Frisch  it  has  been  well  established  that  many  insects  can  avail  themselves  of  this 
polarization  for  navigation  [20-23].  Such  polarization  has  consequences  for  the  segmentation 
of  shadows:  thus,  as  we  will  show  below,  an  area  that  is  inside  a  shadow  cast  by  direct 
sunlight,  but  which  is  lit  by  the  polarized  ambient  sky  light  will  show  a  distinctive 
polarization,  whereas  an  area  that  is  inside  both  the  shadow  cast  by  sunlight  and  the  shadow 
cast  by  skylight  will  show  no  polarization  at  all. 

Because  most  imaging  devices  integrate  light  energy  over  a  time  epoch  that  is  long 
relative  to  the  oscillation  period  (fs),  phase  information  is  not  recorded.  With  the  phase 
information  lost,  when  a  linear  polarization  analyzer  is  placed  in  front  of  the  camera,  the 
measured  intensity  I  at  a  specific  image  location  or  pixel,  as  a  function  of  the  angle  of 
orientation  (p  of  the  polarization  analyzer  is  given  by 
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l(<p)  =  iu+ia  cos[2  {0~(p)\  =  Iu  {l  +  /?  cos  [2(0 -#>)]}, 

where  0  is  the  orientation  angle  of  the  major  axis  of  the  polarization  ellipse,  Iv  is  50%  of  the 
total  intensity  at  each  pixel,  and  p  =  I A  /  Iv  defines  the  degree  of  linear  polarization  at  the 
pixel.  The  reference  axis  for  the  two  angles  (p  and  6  can  be  arbitrarily  chosen,  and  complete 
information  about  the  polarization  state  of  the  light  can  be  obtained  by  capturing  images  with 
the  polarizer  oriented  at  three  different  angles,  for  example  cp  =0,  45  and  90  degrees  [26,35] 
From  these  three  images,  one  can  recover  Iv  ,  IA  ,  and  6  for  each  pixel  of  the  image  using  the 
following  expressions: 

h=(I o  +  4,)/2 

h  =  (2) 

e  =  arctan  [(/45  )]/2 . 

Here  indices  0,  45,  and  90  indicate  the  orientation  of  the  polarizer  in  degrees  when  each 
specific  image  was  taken.  Because  6  and  6+n  are  indistinguishable  for  phase-blind  sensors, 
the  meaningful  range  of  6  is  restricted  to  ?r,  and  6  ranges  from  0  to  n.  In  the  work  presented 
here  we  sample  three  angles  (0,  45  and  90)  by  manually  or  mechanically  rotating  a  single 
linear  polarizer  mounted  in  front  of  an  intensity  integrating  camera.  The  camera  used  in  our 
experiments  is  a  calibrated  Olympus  E-10  digital  camera  with  4  Mega  pixels  and  10  bit  pixel 
depth  (we  use  the  RAW  mode). 

3.  Experiments 

The  first  example  is  an  outdoor  scene  of  a  walkway  in  front  of  a  building  with  all-glass  walls 
(Fig.  2  to  Fig.  4,  the  glass- walled  building  is  visible  in  Fig.  4).  To  help  reader  grasp  the 
relationship  between  pictures  in  Fig.  2  to  Fig.  4,  we  overlay  a  yellow  circle  over  a  sewer 
drainage  cover  that  is  visible  in  all  pictures  but  Fig.  4  Left  to  call  attention  to  readers  that  this 
is  the  exact  same  object  in  all  the  pictures.  Then  a  green  square  is  overlaid  on  the  same  glass 
door  that  is  visible  in  both  Fig.  4  Left  and  Fig.  4  Right.  The  sun  illuminated  the  scene  from  the 
right  hand  side  of  the  picture:  shadows  cast  by  trees  are  seen  along  the  walkway,  most  clearly 
in  the  upper  portion  of  the  image.  Most  existing  shadow  handling  algorithms  would  simply 
segment  the  dark  areas  as  shadows,  reducing  or  eliminating  the  contrast  in  brightness  caused 
by  the  shadow.  However  there  is  a  more  complicated  overlapping  shadow  pattern  hidden 
inside  the  scene  that  is  not  detectable  from  analysis  of  the  intensity  distribution. 
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Fig.  2.  Left:  “Intensity- only”  image  of  an  outdoor  scene  with  light  and  shadow.  Right: 

“Degree- of-polarization”  image  of  the  same  scene;  this  image  plots  the  quantity  p  =  IA/IV  [see 
Eq.  (2)],  extracted  for  each  image  pixel.  Hidden  patterns  of  shadows  within  shadows  are  clearly 
visible  in  high  contrast.  The  glass-walled  building  is  shown  in  Fig.  4.  The  yellow  circle  points 
out  the  same  sewage  drain  cover  that  is  visible  in  all  related  pictures  up  to  Fig.  4  except  Fig.  4 
Left  to  help  orientation. 

In  this  scene  the  glass-wall  building  to  the  left  rear  side  of  the  scene  reflected  sunlight 
from  its  glass  panels,  but  not  from  the  thinner  frames  around  each  piece  of  glass.  The  reflected 
light  was  partially  polarized,  and  the  reflection  pattern  cast  on  the  scene  overlapped  with  the 
shadow  pattern  cast  by  the  direct  sunlight.  The  light  reflected  by  the  glass  was  weaker  than  the 
direct  sunlight,  and  the  pattern  it  creates  is  essentially  invisible  in  the  “intensity-only”  image 
at  left.  However,  when  our  polarization-sensitive  camera  was  used  to  extract  the  “degree  of 
polarization”  image,  the  hidden  pattern  of  overlapping  shadow  was  revealed  (Fig.  2,  right 
panel).  The  area  that  was  lit  neither  by  direct  sunlight,  nor  by  the  reflected  light  from  the 
glass,  is  both  dark  and  unpolarized,  and  thus  appears  dark  in  both  images.  These  are  the  cast 
pattern  of  the  glass  frames  of  the  glass-wall  building  to  the  left  of  the  picture.  The  areas  that 
were  not  lit  by  direct  sunlight  -  and  thus  appear  as  shadows  in  the  intensity-only  image  -  but 
which  were  illuminated  by  the  partially  polarized  reflected  light  from  the  glass-wall  building, 
exhibit  strong  polarization.  The  degree-of-polarization  image  normalizes  the  polarization 
signal  with  respect  to  the  total  intensity  [Eq.  (2)],  so  these  areas  show  up  as  a  bright  pattern  in 
the  degree-of  polarization-image  (Fig.  2,  right).  To  establish  that  this  pattern  is  unique  to  the 
polarization  analysis,  and  not  hidden  in  the  intensity-only  image  due  to  low  contrast  in  the 
shadow  area,  we  performed  linear  contrast  enhancement,  followed  by  gamma  correction  of 
0.5  to  both  images  of  Fig.  2:  from  the  results  (Fig.  3)  it  is  clear  that  the  shadow  patterns  are 
only  revealed  in  the  degree-of-polarization  image.  To  further  document  the  nature  of  the 
sunlight  and  glass  wall  sources  to  the  shadows  revealed  by  polarization  analysis  we  provide 
images  of  the  glass-wall  and  frames  of  the  building  (Fig.  4,  left),  and  of  the  walkway  when  the 
bright  direct  sunlight  is  blocked  (Fig.  4,  right).  Note  that  pictures  in  Fig.  4  are  taken  with  the 
camera  at  about  the  same  position  and  general  view  direction  as  when  the  pictures  in  Fig.  2 
and  Fig.  3  were  taken.  The  only  difference  is  that  in  Fig.  4  the  camera  zooms  out  and  points 
more  upward  in  order  to  put  the  tall  glass-walled  building  into  view.  In  sum,  the  patterns 
revealed  in  the  degree-of-polarization  image  are  indeed  caused  by  shadows  created  by 
polarized  light  reflected  from  the  glass  source. 
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Fig.  3.  Images  in  Fig.  2  are  contrast- enhanced  by  linear  intensity  range  stretch  followed  by 
gamma  correction  of  0.5  to  show  the  details  in  the  dark  area.  Left:  intensity  image.  Right: 
degree  of  polarization  image.  It  is  clear  that  the  pattern  revealed  in  the  polarization  image  is  not 
present  in  the  intensity  image  even  after  contrast  enhancement.  The  yellow  circle  points  out  the 
same  sewage  drain  cover  that  is  visible  in  all  related  pictures  up  to  Fig.  4  except  Fig.  4  Left  to 
help  orientation. 


Fig.  4.  (Pictures  shown  in  this  figure  are  all  regular  intensity  images  with  no  polarization 
information)  Left:  the  glass-wall  building  showing  big  glass  rectangles  and  frames.  Right:  A 
picture  of  the  same  walk  way  as  in  Fig.  2  and  Fig.  3  taken  another  day  when  the  direct  sun  light 
is  blocked  due  to  nearby  construction  scaffolding.  The  shadow  pattern  cast  on  the  walk  way  by 
the  glass-wall  and  frames  is  visible.  The  yellow  circle  in  the  Right  picture  points  out  the  same 
sewer  drain  cover  as  seen  in  Fig.  2  and  Fig.  3.  The  left  and  right  pictures  in  this  figure  can  be 
related  by  the  same  glass  door  bracketed  by  the  overlaid  green  rectangle.  Note  that  pictures  in 
Fig.  4  are  taken  with  the  camera  at  about  the  same  position  and  general  view  direction  as  when 
taking  pictures  in  Fig.  2  and  Fig.  3.  The  only  difference  is  that  in  Fig.  4  the  camera  zooms  out 
and  points  more  upward  in  order  to  put  the  tall  glass- walled  building  into  view. 

We  performed  a  controlled  laboratory  experiment  to  further  confirm  the  results  obtained 
outdoors.  The  setup  comprises  a  150W  incandescent  light  source  illuminating  a  scene  from 
the  direction  opposite  the  camera,  and  a  15W  fluorescent  light  illuminating  the  same  scene 
from  the  direction  corresponding  to  right  hand  side  of  the  picture  (Fig.  5). 
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Fig.  5.  Controlled  lab  experiment  of  complex  overlapping  shadow:  (a)  Overview  of  the 
experiment  setup.  A  metal  pillar  on  a  optical  table  illuminated  by  a  strong  incandescent  light 
from  the  side  opposite  to  the  camera,  while  another  much  weaker  fluorescent  light  illuminating 
from  the  right  hand  side  of  the  picture.  The  polarization  of  the  observed  reflection  from  the  side 
illuminating  fluorescent  light  is  weaker  because  they  are  all  diffusely  scattered  reflection,  as 
opposed  to  the  mostly  Fresnel  reflection  [34]  coming  from  the  incandescent  light  shining 
directly  opposing  the  view  of  the  camera,  (b)  Intensity- only  image,  (c)  Degree- of-polarization 
image. 

In  the  intensity-only  image  only  the  shadow  of  the  knob  cast  by  the  dominant  (incandescent) 
light  source  is  visible.  However,  in  the  degree-of-polarization  image,  additional  information 
is  visible  and  separated  clearly  in  high  contrast.  Specifically,  a  “shadow”  cast  by  the  much 
weaker  fluorescent  light  from  the  right  hand  side  is  revealed  as  a  bright  area  to  the  left  of  the 
metal  pillar.  The  reason  that  this  region  appears  bright  in  the  degree-of-polarization  image  is 
due  to  the  viewing  geometry:  the  strong  light  reflected  from  the  table  is  highly  polarized, 
whereas  the  light  reflected  to  the  camera  from  the  side  source  is  only  weakly  polarized,  and  so 
where  there  is  a  shadow  cast  by  the  weaker  source,  the  degree  of  polarization  is  less  diluted  by 
the  weak  unpolarized  source  and  higher  degree  of  polarization  is  detected.  In  addition,  the 
area  that  is  not  illuminated  by  either  source  is  very  dark  in  the  intensity-only  image  and  is 
least  polarized  and  seen  as  the  darkest  area  in  the  degree-of-polarization  image.  Similarly,  the 
polarization  of  the  image  regions  corresponding  to  areas  lit  by  both  strong  and  weak  light 
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sources  is  lessened  by  the  unpolarized  light  reflected  to  the  camera  from  the  weak  source  at 
the  right  hand  side  of  the  picture.  Segmentation  algorithms  operating  on  the  degree-of- 
polarization  image  can  readily  extract  the  distinctive  “shadow”  cast  by  the  weak  source.  A 
sample  analysis  (Fig.  6  left)  shows  a  segmentation  obtained  by  a  growing  algorithm  that  starts 
with  2x2  regions.  The  side  shadow  area  is  cleanly  separated  from  the  image  when  21  or 
more  regions  are  segmented  (Fig.  6,  right). 


Fig.  6.  Left:  Segmentation  results  from  region- growing  analysis  (starting  with  the  entire  image 
divided  into  2x2  regions  and  with  adjacent  similar  regions  merging  in  each  iteration)  of  Fig.  5 
(c)  into  21  regions.  Right:  Hidden  shadow  area  extracted  from  Fig.  5  (c).  Note  that  this  pattern 
is  only  a  portion  of  a  larger  shadow  of  the  metal  pillar  cast  by  the  source  at  right,  and  that  this 
larger  shadow  is  partially  obscured  by  both  the  small  knob  and  the  shadow  of  the  small  knob 
cast  by  the  source  opposing  the  camera. 

4.  Discussion 

The  processing  of  shadows  in  images  presents  many  difficulties  for  scene  segmentation,  and 
all  existing  methods  for  analyzing  shadows  based  on  intensity-only  information  have 
limitations.  Many  methods  are  designed  for  specific  applications  like  aerial  photography  or 
traffic  monitoring  [1-19],  so  that  the  lighting  condition  is  simplified  or  known  a  priori.  Many 
applications  using  extant  methods  require  a  specific  geometry  of  the  illumination  and  camera, 
and/or  very  precise  calibrations  of  the  pixel  sensitivity  of  the  camera.  The  use  of  polarization 
in  shadow  analysis  and  segmentation  appears  to  be  robust  and  certainly  provides  new  and 
useful  information  that  may  facilitate  segmentation  and  reveal  new  features  of  the  scene  and 
the  sources  that  illuminate  it.  The  polarization  based  shadow  segmentation  method  suggested 
here  does  have  its  own  limitations.  While  this  method  is  not  strictly  tied  to  a  specific  scene 
geometry,  the  method  does  not  work  when  the  scene  signals  happens  to  be  unpolarized 
everywhere,  a  rare  but  possible  scenario.  Nonetheless,  because  signals  extracted  with  Eq.  (2) 
are  strongest  when  there  is  specular  reflection,  the  use  of  the  degree-of-polarization  image  for 
segmentation  can  be  expected  to  give  the  best  results  when  the  source  is  opposite  and  directed 
toward  the  imaging  system.  A  valuable  feature  of  the  method  presented  here  is  that  it  can 
readily  reveal  the  presence  of  multiple  sources  of  widely  varying  “strength”.  As  methods 
have  already  been  developed  for  estimating  the  illumination  directions  of  multiple  light 
sources  from  information  in  the  image  [36-39],  it  can  be  anticipated  that  combining 
polarization  analysis  with  these  methods  will  produce  valuable  new  tools  for  determining  the 
direction  of  illumination  sources.  Investigations  along  these  lines  are  underway.  This  use  of 
polarization  information  in  shadow  detection,  separation  and  contrast  enhancement  will  also 
be  further  enhanced  when  it  is  combined  with  other  well  known  cues  like  intensity,  color,  and 
geometry  to  achieve  more  accurate  shadow  segmentation  and  give  more  detailed  information 
on  the  origin  of  distinct  shadow  components.  While  the  present  investigation  has  based  its 
shadow-segmentation  solely  on  degree-of-polarization  information,  the  additional  cue 
provided  by  the  orientation  of  the  local  polarization  ellipse  [6 in  Eq.  (1)]  can  also  be  used  for 
image  segmentation  (in  much  the  manner  in  which  color  is  used),  and  it  can  also  be 
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anticipated  that  this  will  further  contribute  to  the  method.  Moreover,  as  expected  from  the 
independence  of  polarization  from  the  other  physical  attributes  of  light  and  demonstrated  by 
our  experiments,  information  extracted  by  polarization  about  shadows  is  unique  and  in  general 
cannot  be  extracted  from  other  cues  alone. 

5.  Conclusion 

We  have  presented  a  novel  method  of  shadow  segmentation  based  on  the  local  degree  of 
polarization  in  images  captured  by  a  polarization-sensitive  imaging  system.  This  analysis 
reveals  that  the  polarization  of  light  conveys  distinct  and  valuable  information  about  a  scene 
that  can  be  extracted  at  modest  cost.  Polarization  have  been  used  in  many  other  vision  tasks 
such  as  removing  glare  and  target  detection,  but  to  the  best  of  our  knowledge  has  not 
previously  been  used  to  aid  the  segmentation  of  complex  shadows  in  a  scene.  Polarization 
information  enables  our  system  to  extract  information  about  the  complexities  of  multiple  light 
sources  and  the  overlapping  shadows  they  create.  Such  shadows  are  very  difficult  even  to 
detect  in  intensity-only  images,  and  their  extraction  with  polarization  analysis  provides  a  new 
means  of  identifying  the  direction  and  nature  of  light  sources  illuminating  a  scene. 
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In  forensic  science  the  finger  marks  left  unintentionally  by  people  at  a  crime  scene  are  referred  to  as  latent 
fingerprints.  Most  existing  techniques  to  detect  and  lift  latent  fingerprints  require  application  of  a  certain  ma¬ 
terial  directly  onto  the  exhibit.  The  chemical  and  physical  processing  applied  to  the  fingerprint  potentially  de¬ 
grades  or  prevents  further  forensic  testing  on  the  same  evidence  sample.  Many  existing  methods  also  have 
deleterious  side  effects.  We  introduce  a  method  to  detect  and  extract  latent  fingerprint  images  without  apply¬ 
ing  any  powder  or  chemicals  on  the  object.  Our  method  is  based  on  the  optical  phenomena  of  polarization  and 
specular  reflection  together  with  the  physiology  of  fingerprint  formation.  The  recovered  image  quality  is  com¬ 
parable  to  existing  methods.  In  some  cases,  such  as  the  sticky  side  of  tape,  our  method  shows  unique 
advantages.  ©  2006  Optical  Society  of  America 

OCIS  codes:  150.0150.  110.0110.  100.0100.  260.5430. 


1.  INTRODUCTION 

Fingerprinting  is  one  of  the  most  widely  used  biometric 
methods  for  identifying  and  authenticating  individual 
persons.  The  modern  science  of  fingerprinting  started  in 
the  second  half  of  the  19th  century.  For  an  interesting  his¬ 
torical  review,  see  Refs.  1-3.  There  are  two  types  of  fin¬ 
gerprint  data,  distinguished  by  their  formation  processes. 
In  forensic  science,  finger  marks  left  unintentionally  at  a 
crime  scene  are  referred  to  as  latent  fingerprints.  Finger¬ 
prints  acquired  directly  from  human  fingers  using  ink  or 
scanners  in  controlled  environments  are  referred  to  as  ex¬ 
emplar  fingerprints.1-4  Although  both  types  of  finger¬ 
prints  are  related  to  some  extent,  the  recovery  of  latent 
and  exemplar  fingerprints  poses  very  different  technical 
challenges.  There  have  been  considerably  more  successful 
optical  methods  for  exemplar  fingerprints  than  for  latent 
fingerprints,  and  sometimes  the  classification  can  be  con¬ 
fusing.  For  example,  there  exist  methods  that  use  laser 
and  polarization  for  extracting  fingerprint  images  directly 
from  live  human  fingers5;  such  methods  are  for  acquiring 
exemplar  fingerprints.  The  purpose  and  the  detailed 
physical  background  of  that  system  are  different  from 
those  involved  in  the  application  of  laser  or  polarization  to 
latent  fingerprints.  Another  example  is  a  device  that 
claims  that  it  “optically  reads  a  latent  fingerprint.”6  How¬ 
ever,  the  main  function  of  the  particular  device  is  to  read 
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directly  from  a  human  finger  (and  simultaneously  com¬ 
pare  with  a  known  fingerprint  pattern),  and  such  a  device 
should  be  classified  as  an  exemplar  fingerprint  reader. 
Recovery  of  latent  fingerprints  is  much  more  difficult 
than  the  recovery  of  exemplar  fingerprints  because  the 
physical  and  chemical  composition  of  latent  fingerprints 
and  the  surfaces  on  which  they  are  found  vary  greatly 
and  can  often  undergo  unknown  degradation  before  being 
examined.  In  this  paper  we  present  a  new  method  for  the 
detection  and  recovery  of  latent  fingerprints. 

Latent  fingerprints  differ  from  exemplar  fingerprints  in 
that  they  are  very  difficult  to  detect  with  unaided  human 
vision  under  most  ordinary  viewing  conditions  (hence 
their  name);  they  are  usually  also  of  lower  quality  than 
exemplar  fingerprints,  although  high-quality  fingerprint 
marks  can  at  times  be  found  at  a  crime  scene.  To  be  pre¬ 
cise,  nonexemplar  fingerprints  that  can  be  easily  seen  by 
a  human  observer  should  be  called  patent  fingerprints.1-4 
In  practice,  however,  the  term  latent  fingerprints  is  often 
used  to  refer  to  all  fingerprints  that  are  not  exemplar.  It  is 
really  latent  fingerprints  that  are  more  common  at  a 
crime  scene  and  require  greater  efforts  to  render  visible. 
Most  techniques  employed  for  this  purpose  utilize  a 
chemical  or  physical  process  that  applies  some  kind  of 
material  directly  to  the  surface  suspected  to  bear 
fingerprints.1-4,7-9  Once  the  contrast  of  the  fingerprint 
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mark  is  sufficiently  enhanced  by  such  treatments,  the 
mark  is  either  photographed  or  lifted  to  be  permanently 
archived  as  evidence.  The  term  lifting  the  fingerprint 
originates  from  the  oldest,  but  still  widely  used,  finger¬ 
print  detection  method — powdering — in  which  the  pow¬ 
ders  applied  adhere  to  the  fingerprint  material,  and  then 
are  physically  lifted  out  of  the  original  crime  scene  object 
by  sticky  tape. 

Since  applying  chemicals  or  powders  onto  a  surface  on 
which  fingerprints  reside  changes  the  chemical  and/or 
physical  composition  of  the  surface,  the  use  of  such  inva¬ 
sive  methods  can  potentially  interfere  with  subsequent  fo¬ 
rensic  testing  of  different  types,  and  can  sometimes  inflict 
deleterious  side  effects  on  the  surface  or  the  operator. 
Therefore,  in  the  past  30  years  several  techniques  that 
can  recover  latent  fingerprints  without  the  need  to  apply 
foreign  material  directly  onto  the  fingerprints  have  been 
developed.  Many  of  these  methods  use  specialized  light 
sources  (e.g.,  laser,  UV),  filters,  and  detectors.1-4’7-12  They 
are  successful  in  some  cases.  However,  like  all  other  ex¬ 
isting  techniques,  they  do  not  work  in  all  possible  cases 
and  are  known  to  fail  completely  with  certain  types  of  la¬ 
tent  fingerprints  or  object  surfaces.  As  a  result,  chemical 
enhancers  are  often  reintroduced  to  aid  in  the  detection  of 
latent  fingerprints.1-3,9’12-19  Further  studies  show  that 
techniques  using  special  light  sources  and  filters  work 
much  better  and  in  more  cases  when  combined  with  the 
application  of  certain  chemicals  on  the  fingerprint  sample 
first.1’9’12-19  However,  the  application  of  chemicals  di¬ 
rectly  onto  the  fingerprints  effectively  negates  the  advan¬ 
tages  of  noncontact  methods,  and  the  composite  methods 
revert  back  into  invasive. 

It  has  been  known  by  experienced  law  enforcement  of¬ 
ficers  that  by  varying  the  angle  of  a  flashlight  shining 
onto  a  surface  suspected  to  bear  latent  fingerprints,  one 
can  potentially  locate  latent  fingerprints  that  are  other¬ 
wise  difficult  to  see.7  However,  to  lift  the  latent  finger¬ 
prints  in  a  form  that  can  be  documented  and  presented  as 
evidence  in  a  court  trial,  some  invasive  enhancement 
treatments  are  usually  considered  necessary.  Pfister9’20 
devised  an  optical  method  that  uses  a  semitransparent 
mirror  that  can  project  light  onto  a  surface  at  a  right 
angle,  while  at  the  same  time  allowing  a  camera  or  ob¬ 
server  to  view  the  surface  at  a  right  angle.  A  smooth  sur¬ 
face  is  expected  to  appear  bright  due  to  strong  specular 
reflection,  while  a  fingerprint  mark  would  appear  darker 
due  to  much  less  specular  reflection.  Margot  and 
Lennard9’21  reported  that  such  a  method  works  better 
when  the  sample  is  pretreated  with  cyanoacrylate.  The 
use  of  only  a  right  angle  in  Pfister’s  method  sacrificed 
quite  a  bit  of  flexibility,  and  polarization-based  techniques 
cannot  be  applied  to  further  improve  the  contrast  because 
the  specular  reflection  observed  at  a  right  angle  from  the 
surface  is  not  preferentially  polarized.  It  is  widely  known 
that  specularly  reflected  light  from  dielectrics  would  be 
partially  polarized  at  a  certain  range  of  viewing  angles. 
However,  to  the  best  of  our  knowledge,  no  known  applica¬ 
tion  of  polarization  has  been  reported  for  latent  finger¬ 
prints,  with  the  exception  of  using  a  polarizing  filter  to  re¬ 
move  glare  when  taking  pictures,  which  is  considered  a 
standard  photographic  technique.  Menzel22  briefly  men¬ 
tioned  the  possibility  of  using  optical  polarization  to  en¬ 
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hance  visibility  of  latent  fingerprints  left  on  glass,  but  it 
appears  that  no  further  development  took  place. 

We  propose  here  a  new  method  that  allows  the  detec¬ 
tion  and  lifting  of  latent  fingerprints  into  clearly  identifi¬ 
able  digital  images  without  the  application  of  chemical 
treatments  or,  indeed,  without  any  physical  contact  with 
the  surface  and  fingerprint  material.  Rather  than  employ¬ 
ing  extraneous  material,  our  method  takes  advantage  of 
the  optical  properties.  In  particular  we  exploit  those  prop¬ 
erties  related  to  specular  reflection  and  polarization  of  the 
latent  fingerprint,  which  usually  consists  of  tiny  ridges  of 
skin  residue  material  including  sweat  (salty  water), 
grease,  and  lipid,2’3’23  all  of  which  are  rather  transparent 
dielectric  materials,  making  them  difficult  to  detect  under 
most  viewing  conditions.  Our  method  is  also  applicable  to 
latent  fingerprints  left  on  a  smooth  but  pliable  dielectric 
surface.  The  recovered  fingerprint  images  have  compa¬ 
rable  or  better  quality  than  those  obtained  by  conven¬ 
tional  methods. 


2.  NONCONTACT  OPTICAL  LATENT 
FINGERPRINT  ENHANCEMENT  AND 
LIFTING:  CORE  TECHNIQUE  OUTLINE  AND 
EXPERIMENTAL  SETUP 

We  start  with  the  principal  physical  and  physiological  ba¬ 
sis  of  our  noncontact  optical  latent  fingerprint  enhance¬ 
ment  and  lifting  in  this  section.  The  actual  formulas  used 
in  our  computation  will  follow  later.  The  physics  underly¬ 
ing  the  method  is  illustrated  in  Fig.  1.  When  a  finger 
touches  the  object  surface,  a  dielectric  residue  mark  bear¬ 
ing  the  fingerprint  pattern  is  imprinted  on  it.  The  residue 
on  the  surface  induces  differences  in  optical  polarization 
or  reflection  or  both  between  the  clean  part  of  the  surface 
and  that  bearing  the  print.  The  optical  information  is  cap¬ 
tured  and  enhanced  by  our  unique  optical  setup  and 
stored  as  digital  images.  Further  digital  processing  of  the 
captured  images  enables  us  to  develop  or  lift  the  latent 
fingerprint  pattern  without  applying  any  powder  or  other 
chemicals  to  the  object.  Our  optical  setup  is  based  on  the 
well-known  Fresnel  reflection  theory  for  orthogonal  polar¬ 
izations  and  the  theories  for  macroscopic  surface  reflec¬ 
tance  developed  for  computer  vision  and  graphics.  As  an 
aside,  it  is  interesting  to  note  that  biologists  and  zoolo¬ 
gists  have  found  that  certain  animal  species  have  visual 
systems  that  sense  and  utilize  (in  or  near)  visible  light’s 
polarization  in  the  natural  environment,  e.g.,  backswim- 
mer  Notonecta  glauca  can  detect  the  polarization  of  light 
reflected  from  smooth  water  surfaces  and  use  it  to  land 
and  plunge  safely  on  the  water  surface.24’25  Indeed,  our 
original  step  to  design  our  optical  setup  for  latent  finger¬ 
print  detection  was  inspired  from  this  ability  of  Notonecta 
glauca  in  detecting  the  surface  of  the  water. 

In  Fig.  1(c),  we  illustrate  a  cross-section  view  of  the  fin¬ 
gerprint  on  a  surface.  The  ridge  area  corresponds  to  a 
small  amount  of  residue  on  the  surface,  while  the  furrow 
area  does  not.  All  existing  enhancement  methods  take  ad¬ 
vantage  of  this  situation  by  applying  materials  that  selec¬ 
tively  attach  to  or  interact  with  only  the  residue  area  and 
produce  a  colored  or  fluorescent  pattern  of  the  residue 
area.  Our  noncontact  method  exploits  this  situation  in  a 
different  way  [Fig.  1(d)],  with  a  common  household  light 
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Fig.  1.  (Color  online)  Schematic  of  the  physical  principles  concerning  noncontact  latent  fingerprint  enhancement  and  lifting,  (a)  Mac¬ 
roscopic  reflection  from  a  surface  consists  of  two  distinct  kinds,  i.e.,  specular  and  diffuse.  For  specular  reflection  the  angles  of  incidence 
and  reflection  are  equal,  while  for  the  diffuse  case  the  reflected  intensity  may  approximately  have  an  effectively  uniform  distribution  over 
all  directions  in  a  hemisphere.  Most  surfaces  exhibit  both  types  of  reflection,  but  one  type  may  be  stronger  than  the  other,  (b)  Partial 
polarization  of  specularly  reflected  light  from  a  semitransparent  dielectric  surface.  It  is  known  that  the  light  reflected  from  the  smooth 
surface  is  partially  polarized  with  the  polarization  being  perpendicular  to  the  plane  of  reflection,  (c)  Live  human  skin  is  kept  soft  and 
pliable  by  the  constant  oily  secretion  of  hypodermic  glands.  The  ridge  area  of  the  skin  pattern  tends  to  leave  a  dielectric  residue  on  a 
surface  touched  by  a  finger,  (d)  The  residue  left  in  (c)  forms  the  latent  fingerprint.  Using  a  method  that  generates  a  sufficient  contrast 
difference  between  the  latent  fingerprint  and  the  rest  of  the  surface  in  the  camera  image,  a  successful  detection  and  extraction  can  be 
achieved  without  applying  physical  or  chemical  treatments  to  the  surface.  Note  that  the  camera  position  in  (d)  is  oriented  in  a  such  a  way 
that  it  captures  the  specular  component  of  reflected  light  from  the  clean  surface  only  while  the  specular  reflection  component  of  the 
residue  is  not  captured.  Additionally,  when  a  finger  touches  a  pliable  dielectric  surface  it  could  cause  a  plastic  print  on  the  surface.  In  this 
case  the  difference  in  the  surface  normal  caused  by  the  plastic  print  ridges  will  serve  the  same  purpose  of  creating  contrast  in  intensity 
and  polarization  under  proper  lighting. 


source  (incandescent  or  fluorescent,  does  not  really  mat¬ 
ter  here),  a  camera,  and  the  surface  being  inspected  and 
arranged  in  such  a  way  that  the  geometry  conforms  ap¬ 
proximately  to  the  law  of  (specular)  reflection.  Thus,  the 
incident  angle  of  light  from  the  source  approximately 
equals  the  viewing  (reflection)  angle  of  the  camera,  so 
that  the  camera  will  capture  the  light  reflected  specularly 
from  the  nonresidue  area,  and  also  only  the  light  reflected 
diffusely  from  the  residue-laden  area.  The  reason  for  this 
arrangement  is  that  the  residue  stain  area  is  likely  to 
have  different  surface  normal  directions  and  indices  of  re¬ 
fraction  as  compared  with  the  uniform  or  smoothly  vary¬ 
ing  surface  normal  direction  of  the  unstained  surface 
area.  The  localized  nature  of  specular  reflection  energy 
distribution  makes  it  sensitive  to  changes  in  the  direction 
of  the  surface  normal  caused  by  the  presence  of  finger¬ 
print  residue  on  the  surface.  Since  the  specular  reflection 
component  is,  in  general,  much  stronger  than  the  diffuse 
reflection  component26  [see  Fig.  1(a)],  one  potentially 
finds  an  enhanced  contrast  between  the  residue-laden 
ridge  mark  and  the  clean  surface  furrow  negative  mark. 
Another  often-encountered  case  is  a  plastic  fingerprint 
left  on  a  pliable  dielectric  surface.  There  may  or  may  not 
be  biological  residue  left  on  the  surface,  but  the  ridge  and 


furrow  patterns  formed  by  the  pliable  dielectric  material 
itself  create  differences  in  surface  normal  compared  with 
the  undisturbed  surface  area  and  will  serve  the  same  pur¬ 
pose. 

Note  that  Fig.  1(d)  is  not  drawn  to  scale.  The  micro¬ 
structure  of  the  fingerprint  ridges  has  been  magnified 
hundreds  of  times  for  illustration  purposes  and  the  sizes 
of  the  light  source  and  the  camera  and  the  distances  from 
each  of  them  to  the  sample  surface  have  been  greatly 
scaled  down  to  fit  in  the  limited  figure  space.  In  Fig.  1,  if 
one  tries  to  draw  straight  lines  linking  a  point  on  the  light 
source  to  a  ridge  and  another  straight  line  from  the  same 
point  on  the  light  source  to  a  valley  point  in  Fig.  1(d),  it 
would  appear  that  the  two  lines  are  far  from  parallel.  The 
fact  is  that  it  does  not  make  any  sense  to  draw  these  two 
lines  in  Fig.  1(d)  where  sizes  and  distances  were  not 
drawn  to  scale.  A  typical  fingerprint  is  quite  small,  ~  1  in. 
(25.4  mm)  by  0.3-0. 5  in.  (7.6-12.7  mm)  while  a  typical 
light  bulb  is  ~3  in.  (76  mm)  in  diameter  and  needs  to  be 
placed  at  least  7  in.  (177.8  mm)  from  the  sample  surface 
to  avoid  blocking  the  field  of  view  of  the  observing  camera 
or  overheating  the  surface.  In  Fig.  2  we  show  a  drawing 
that  is  more  to  scale.  The  area  occupied  by  the  print  mark 
of  one  finger  extends  to  only  the  small  area  around  the 
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point  P  in  Fig.  2.  There  are  typically  more  than  50  ridges 
and  valley  periodic  patterns  within  this  small  area.  The 
distance  from  a  ridge  point  to  its  immediate  adjacent  val¬ 
ley  point  is  typically  only  —0.5  mm.  Note  that  we  only 
need  to  make  sure  that  the  contrast  between  one  valley 
and  its  immediate  adjacent  ridge  area  is  high  enough  for 
detection.  So  the  only  thing  we  need  to  check  is  whether 
light  coming  from  the  same  point  source  would  have  prac¬ 
tically  the  same  incident  direction  for  a  point  on  a  finger¬ 
print  ridge  area  compared  with  a  point  on  the  valley  area 
immediately  adjacent  to  the  ridge.  This  is  indeed  the  case. 
For  Fig.  2,  if  we  have  typical  setup  dimensions  of  OP 
=  177.8  mm  and  OL=  177.8  mm,  with  an  incident  angle  of 
45  deg,  then  the  incident  angles  of  light  from  the  same 
source  point  L  to  the  immediate  adjacent  valley  points 
(suppose  that  P  is  a  ridge  point)  are  arctan[(177.8 
+  0.5)/177.8]  =  45.08  deg  and  arctan[(177.8-0.5)/177.8] 
=  44.92  deg,  respectively.  They  are  indeed  practically  par¬ 
allel.  All  realistic  light  sources  are  extended  light  sources 
that  can  be  modeled  as  a  group  of  a  countless  number  of 
point  sources.  The  illumination  effects  can  be  found  by  in¬ 
tegrating  the  contribution  from  every  light-emitting  point 
source  comprising  the  whole  light  source.  Since  we  are  us¬ 
ing  an  ordinary  incoherent  light  source,  the  contributions 
to  the  irradiance  from  each  point  source  simply  add  up 
and  we  do  not  need  to  consider  interference  here.  Figures 
2(a)  and  2(b)  also  describe  how  the  directional  reflecting 
nature  of  specular  reflection  and  the  directional  light  ac¬ 
ceptance  of  the  observing  camera  work  together  to  create 
the  desired  contrast  between  adjacent  fingerprint  ridge 
and  valley  areas  using  a  simple  noncollimated  extended 
light  source  like  a  light  bulb  with  a  diffuser  plate.  Be¬ 
cause  a  camera  placed  at  point  C  records  only  light  energy 
reflected  from  P  along  the  direction  PC,  only  light  inci¬ 
dent  along  the  direction  LP  would  have  almost  all  its 
specular  reflection  component  being  recorded  by  the  cam¬ 
era  at  point  C  for  the  specular  irradiance  (7S)  of  point  P. 
The  diffuse  reflection  components  have  energy  almost 
evenly  spread  in  all  directions  and  only  those  small  frac¬ 
tions  of  energy  directed  toward  the  direction  LP  would  be 
recorded  by  the  camera  at  C  for  the  diffuse  reflection  ir¬ 
radiance  ( ID )  for  point  P.  In  Fig.  2(a),  if  we  set  up  colli¬ 
mated  light  beams,  then  every  incident  light  beam  would 
contribute  about  the  same  ratio  of  specular  irradiance 
and  diffuse  reflection  irradiance.  In  Fig.  2(b)  we  show 
what  happens  when  a  noncollimated  extended  light 
source  is  used.  First,  while  there  can  be  many  incident 
light  rays  parallel  to  the  direction  LP,  they  do  not  contrib¬ 
ute  to  the  specular  reflection  irradiance  of  point  P  in  the 
observed  image  because  the  camera  at  point  C  does  not 
see  them  reflecting  from  the  point  P,  except  only  the  ray 
LP.  Any  light  ray  incident  from  a  direction  different  from 
the  direction  LP  will  have  its  specular  reflection  compo¬ 
nent  reflected  to  directions  other  than  the  direction  PC, 
but  the  camera  only  collects  irradiance  energy  emitted 
along  the  direction  PC  for  the  point  P,  so  this  has  no  effect 
on  our  images.  The  light  rays  coming  from  different 
angles  do  contribute  to  the  diffuse  reflection  component 
because  diffuse  reflection  energy  is  almost  evenly  distrib¬ 
uted  along  all  directions  regardless  of  the  incident  direc¬ 
tion.  Since  the  recorded  irradiance  is  the  algebraic  sum  of 
all  collected  irradiance,  the  observed  specular  reflection 


irradiance  would  come  from  only  the  incident  light  along 
the  direction  of  LP.  Thus  the  specular  reflection  signal  Is 
remains  the  same  for  both  Figs.  2(a)  and  2(b).  The  diffuse 
reflection  signal  Id  would  be  stronger  in  Fig.  2(b)  than  in 
2(a).  Thus  the  effect  of  using  a  noncollimated  extended 
light  source  is  just  some  decrease  in  contrast  between  Is 
and  Id •  Note  that,  although  ID  receives  contribution  from 
more  light  source  points  than  Is ,  each  contribution  is 
rather  small  compared  with  the  strength  of  the  specular 
reflection.  Thus  in  many  cases  we  can  still  observe  the  fin¬ 
gerprint  pattern  easily.  Note  also  that  when  the  distance 
S  is  large  compared  with  the  dimension  of  the  light 
source,  then  the  incident  angle  differences  a  will  be  neg¬ 
ligible.  In  other  words,  an  ordinary  household  light  source 
like  a  lamp  can  nicely  approximate  an  ideal  point  light 
source  as  long  as  the  distance  S  is  several  times  that  of 
the  dimension  of  the  light  source. 

Another  point  we  want  to  mention  here  is  the  use  of  a 
diffuser  in  front  of  the  light  bulb  in  our  experiment. 
Specular  reflection  is  the  same  type  of  reflection  we  see 
from  a  mirror,  which  means  that  by  observing  the  specu¬ 
lar  reflection  we  will  be  seeing  the  mirror  image  of  our 
light  source.  A  bare  light  bulb  without  a  diffuser  will 
project  its  own  image,  e.g.,  the  filament  and  its 
shade,...  etc.,  on  top  of  the  fingerprint  sample.  This  would 
add  clutter  to  the  fingerprint  image  we  want  to  recover. 
Adding  a  diffuser  in  front  of  a  light  bulb  would  diffuse  the 
mirror  image  pattern  of  the  light  source  so  that  we  get  a 
cleaner  fingerprint  image.  The  image  of  a  lighted  diffuser 
plate  is  almost  featureless,  so  its  image  projected  on  the 
surface  would  not  add  its  own  patterns  over  potential  fin¬ 
gerprint  patterns  on  the  surface  being  illuminated.  One 
should  not  confuse  this  diffuser  with  the  term  diffuse  re- 


Fig.  2.  (Color  online)  Drawings  depicting  the  same  setup  as  in 
Fig.  1(d)  but  with  relative  distances  and  object  sizes  drawn  to 
scale  to  explain  the  effects  of  a  real  light  source  compared  with 
an  idealized  collimated  light  source.  In  both  (a)  and  (b),  L  is  the 
position  of  the  light  source;  O  is  the  orthogonal  projection  on  the 
surface  for  L;  P  is  a  ridge  or  valley  point  around  the  center  of  the 
fingerprint  pattern;  C  is  the  camera  view  point.  ID  is  the  irradi¬ 
ance  of  the  point  P  from  diffuse  reflection  and  recorded  by  the 
camera  at  C.  It  is  typically  weak  and  is  also  represented  here  as 
the  small  arrow(s)  along  the  direction  PC.  Is  is  the  irradiance  of 
the  point  P  from  specular  reflection  and  recorded  by  the  camera 
at  C.  It  is  typically  much  stronger  compared  with  irradiance  from 
diffuse  reflection,  and  is  depicted  here  by  the  large  arrow  along 
the  direction  PC.  (a)  The  simple  condition  when  the  light  source 
is  effectively  collimated  along  the  direction  of  LP.  (b)  The  situa¬ 
tion  for  when  a  noncollimated  extended  light  source  is  used.  The 
Is  remains  the  same  as  in  (a)  but  ID  is  stronger  due  to  the  con¬ 
tribution  from  more  point  sources.  The  end  result  is  decreased 
contrast  between  Is  and  Id •  However,  since  only  Is  contains  a  po¬ 
larized  component,  the  polarization-based  method  is  equally  ef¬ 
fective  in  both  conditions. 
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flection  we  mentioned  elsewhere  in  this  paper.  Diffuse  re¬ 
flection  and  specular  reflection  are  two  different  types  of 
reflection  from  a  surface  (although  in  reality  they  coexist 
in  virtually  every  surface  reflection).  A  surface  reflects 
any  incident  light  partially  in  the  form  of  diffuse  reflec¬ 
tion  and  partially  in  the  form  of  specular  reflection,  re¬ 
gardless  of  how  and  where  the  incident  light  comes  from. 
In  other  words,  any  incident  light,  whether  it  is  directly 
from  a  light  bulb  or  has  undergone  scattering  by  a  dif¬ 
fuser,  will  be  reflected  by  the  surface  we  are  examining 
partially  via  specular  reflection  and  partially  via  diffuse 
reflection.  The  diffuser  we  put  in  front  of  the  light  bulb 
has  nothing  to  do  with  the  surface  on  which  we  look  for 
fingerprints.  The  diffuser  does  not  even  have  anything  to 
do  with  reflection  on  the  fingerprint-bearing  surface  that 
is  the  main  topic  of  our  detection  method.  The  function  it 
performs  in  our  setup  is  scattering  and  transmitting  the 
light  before  it  reaches  the  surface-bearing  fingerprint. 

The  trade-off  of  using  a  diffuser  is  a  stronger  observed 
diffuse  reflection  for  every  point,  and  thus  reduced  finger¬ 
print  contrast  in  the  specular-reflection-only-based 
method.  In  some  cases  the  contrast  between  the  diffuse 
reflection  component  and  the  specular  reflection  effect  be¬ 
comes  too  low  to  be  useful.  It  is  also  possible  that  the  ob¬ 
ject  itself  may  have  a  complicated  high-contrast  pattern 
under  the  top  coating  of  the  surface  [see  Figs.  1(c)  and 
1(d)]  that  interferes  with  the  fingerprint  pattern  even  af¬ 
ter  the  enhancement.  This  problem  has  been  recognized 
by  many  practitioners.27,28  Discrete  Fourier- transform 
analysis  has  been  shown  to  be  able  to  remove  regular  pat¬ 
terns  that  vary  periodically,  but  cannot  deal  with  a  gen¬ 
eral  background  that  is  not  periodic.  In  cases  where  the 
intensity  difference  caused  by  specular  reflection  alone  is 
not  enough  to  detect  fingerprints,  we  use  polarization  im¬ 
aging  to  accomplish  the  fingerprint  detection.  An  addi¬ 
tional  characteristic  of  specular  reflection  is  that  it  tends 
to  be  partially  polarized  in  a  plane  perpendicular  to  the 
plane  of  reflection  [see  Fig.  1(b)] .  One  or  more  polarization 
analyzers  collecting  polarization  components  at  different 
angles  can  provide  complete  information  about  the  polar¬ 
ization  state  of  the  reflected  light.  On  the  basis  of  the  po¬ 
larization  information,  we  can  further  extract  only  the 
specular  component  of  this  reflection  and  get  a  much 
cleaner  fingerprint  image  because  for  the  most  part  the 
light  coming  from  the  pattern  beneath  the  top  coating  of 
the  object  surface  is  due  mostly  to  unpolarized,  diffuse  re¬ 
flection.  We  have  also  found  that  in  some  cases  some  of 
the  polarization  images  simply  show  higher  contrast  be¬ 
tween  the  fingerprint  and  its  background  than  the  finger¬ 
print  images  recovered  using  specular  reflection  alone. 

The  general  expression  for  the  observed  intensity  of 
partially  polarized  light  7  as  a  function  of  the  angle  <p  of 
orientation  of  a  polarization  analyzer  can  be  written  as 
follows29’30: 

I(<p)=Iu  +  IA  cos[2(6>-  <p)]  =Iu{l  +p  cos[2(6>-  <p)]},  (1) 

where  6  is  the  orientation  angle  of  the  major  axis  of  the 
polarization  ellipse,  Ijj  is  half  of  the  total  pixel  intensity, 
and  p=1a^u  is  the  degree  of  linear  polarization  at  a 
given  pixel  in  a  digitized  image.  The  reference  axis  for  cp 
and  6  can  be  arbitrarily  chosen.  Since  there  are  more  than 
one  unknown  parameters,  putting  one  polarizer  at  a 


Fig.  3.  (Color  online)  Example  experimental  setup  overview  for 
Fig.  5. 


Fig.  4.  Picture  of  the  three  sample  items  bearing  latent  finger¬ 
prints:  a  hardcover  book,  a  plastic  CD  case  with  underlying  in¬ 
sert  patterns,  and  a  stainless  steel  blade  of  a  Swiss  army  knife. 
Experimental  results  on  these  items  are  presented  in  Figs.  8-10. 

given  orientation  angle  in  front  of  the  camera  and  taking 
a  picture  cannot  provide  complete  information  about  the 
polarization  state  of  the  received  light.  By  taking  three 
pictures  with  the  polarizer  oriented  at  three  different 
angles,  for  example,  cp= 0,  45,  and  90  deg,  we  can  recover 
lu,  I  a,  and  6  for  each  pixel  of  the  image  using  the  follow¬ 
ing  expressions: 


Iu=(I0  +  I90)/2, 

I  A  ~  \Z(^45  -III)2  +  (Iu~  1 9o)2  j 


6  =  arctan[(/45  -  I^/ilu  -  790)]/2 .  (2) 

Here  indices  0,  45,  and  90  indicate  the  orientation  of  the 
polarizer  in  degrees  when  the  image  was  taken.  Because  6 
and  0+  77  are  indistinguishable  for  phase-blind  visual  sen¬ 
sors  in  most  conventional  cameras,  the  meaningful  range 
of  6  is  restricted  to  (0  —  77).  We  usually  use  6  in  the  range 
from  0  to  77.  Polarization  camera  systems  able  to  rapidly 
take  the  required  pictures  have  been  developed  by  Wolff 
and  his  colleagues.31-33  The  formulation  and  symbols 
used  here  follow  from  our  previous  work29,30  and  are 
slightly  different  from  those  used  by  Wolff.  Since  the 
background  object  pattern  is  most  likely  caused  by  pig¬ 
ments  beneath  the  transparent  substrate  that  is  used  to 
hold  them,  the  object  pattern  intensity  signals  are  mostly 
due  to  diffuse  reflection,  which  is  nearly  unpolarized,  and 
thus  I  a  and  sometimes  p  are  close  to  zero.  Thus,  with  our 
polarization  technique  we  can  extract  the  purely  specular 
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(g)  (h) 

Fig.  5.  Fingerprint  detection  experiment  on  a  sample  surface,  a  metal  case  of  a  pump  painted  orange,  (a)  Sample  surface  picture  taken 
under  ordinary  lighting,  linearly  scaled,  (b)  Same  as  (a),  but  by  histogram  equalization  for  contrast  enhancement,  (c)  The  same  surface 
as  (a)  and  (b)  taken  under  our  special  lighting  setup  in  which  the  clean  surface  without  fingerprint  residue  is  showing  strong  specular 
reflection.  The  fingerprint  residue  disrupts  the  specular  reflection  geometry  so  its  pattern  is  revealed  as  a  dark  diffuse  reflection  pattern. 
This  image  is  linearly  rescaled,  (d)  Same  as  (c),  only  the  contrast  enhancement  is  done  using  histogram  equalization,  (e)  Zoom-in  view  of 
the  fingerprint  revealed  in  (c)  and  (d).  (f)  Further  zoom-in  view  of  (e),  showing  the  very  fine  details  of  the  recovered  fingerprint  pattern, 
(g)  The  same  fingerprint  being  lifted  with  tape  after  being  dusted  with  forensic  black  magnetic  powder,  (h)  The  fingerprint  lifted  using 
the  traditional  powdering  and  tape  lifting.  The  fingerprint  lifted  by  the  proposed  new  method  as  shown  in  (e)  and  (f)  is  cleaner. 
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reflection  component  from  the  top  surface  by  computing 
images  of  IA  or  p  for  every  image  point.  Such  images  often 
carry  a  substantial  contrast  between  the  fingerprint  resi¬ 
due  pattern  and  the  clean  area  in  between.  Note  that  if 
only  0  and  90  deg  orientation  images  are  taken,  the  fin¬ 
gerprint  may  still  be  enhanced  in  the  polarization- 
difference  image,34-38  but  the  0  or  90  deg  direction  must 
be  nearly  parallel  to  either  the  object  surface  or  the  fin¬ 
gerprint  ridge  surface,  which  can  be  challenging  to  ar¬ 
range  when  the  surface  is  not  flat.  Figure  3  is  a  view  of 
our  experimental  setup  that  is  arranged  according  to  the 
geometry  shown  in  Fig.  1(d).  Figure  4  shows  some  of  our 
test  items  that  are  often  found  to  bear  fingerprints. 

3.  EXPERIMENTS 

A  step-by-step  application  of  the  new  optical  method  is 
presented  in  Fig.  5.  Figure  3  gives  an  overview  of  the  ex- 


(c) 

Fig.  6.  Paper  calendar  cover  with  underlying  picture,  (a)  A  fin¬ 
gerprint  is  revealed  by  specular  lighting,  (b)  Same  item  as  in  (a) 
but  with  polarization  processing.  This  is  the  IA  image;  back¬ 
ground  is  completely  removed,  (c)  Zoom-in  view  of  the  fingerprint 
in  (b). 
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(d) 

Fig.  7.  Soft  plastic  CD  sleeve  with  white  cotton  lining  under¬ 
neath.  (a)  Under  ordinary  lighting,  (b)  Polarization  IA  image.  The 
latent  fingerprint  on  the  CD  sleeve  is  exposed  in  high  contrast, 
(c)  Zoom-in  view  of  the  fingerprint  in  (b).  (d)  The  periodic  pattern 
caused  by  the  cotton  lining  as  seen  in  (c)  can  be  removed  by 
Fourier-transform  processing. 

perimental  setup  when  the  specially  arranged  light  is  on 
and  the  ordinary  room  light  is  turned  off.  We  have  per¬ 
formed  the  experiments  with  both  the  room  light  on  and 
off  and  found  that  the  results  are  very  similar.  This 
means  that  our  method  can  be  easily  applied  to  a  crime 
scene  without  strict  ambient  lighting  control.  The  surface 
being  inspected  is  the  metal  casing  of  an  electric  air  pump 
(Linicon),  which  is  painted  orange.  Figures  5(a)  and  5(b) 
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show  how  the  surface  looks  with  the  specially  arranged 
lighting  turned  off  and  the  ordinary  diffuse  fluorescent 
room  light  on.  The  image  in  Fig.  5(a)  is  shown  with  a  com¬ 
mon  digital  enhancement  available  in  most  image- 
processing  software,  setting  the  brightest  pixels  in  the  im¬ 
age  to  the  maximum  possible  value  allowed  by  the 
display,  and  the  darkest  pixels  to  the  lowest  display  value 
and  linearly  rescaling  the  rest  of  the  pixel  intensity  val- 


(c) 


Fig.  8.  (a)  Close-up  view  of  the  hardcover  book  bearing  a  latent 

fingerprint  under  normal  (no  polarizer,  no  special  lighting  ar¬ 
rangements)  viewing  conditions.  Note  that  this  image  has  under¬ 
gone  digital  linear  contrast  enhancement  but  the  fingerprint 
mark  is  still  not  visible,  (b)  The  same  area  as  in  (a)  but  taken 
with  our  specially  arranged  specular  lighting  condition.  The  la¬ 
tent  fingerprints  are  revealed,  (c)  The  same  area  as  in  (a)  but 
taken  with  our  specially  arranged  specular  lighting  condition 
plus  polarization  image  processing.  This  is  the  IA  image  linearly 
rescaled  to  fit  an  8  bit  display.  The  latent  fingerprints  are  re¬ 
vealed,  and  at  the  same  time  the  background  pattern  from  the 
book  title  is  greatly  suppressed. 
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Fig.  9.  (a)  Close-up  view  of  the  plastic  CD  case  with  insert  pat¬ 

tern  under  normal  viewing  conditions.  No  fingerprint  is  visible 
although  the  image  has  undergone  digital  linear  contrast  en¬ 
hancement.  (b)  The  same  area  of  the  plastic  CD  cover  with  the 
insert  pattern  as  in  (a),  but  taken  with  our  specially  arranged 
specular  lighting  condition  with  three  different  polarizer  orienta¬ 
tions  and  then  the  degree  of  polarization  image  computed.  The 
latent  fingerprints  are  revealed,  (c)  The  same  area  of  the  plastic 
CD  cover  with  the  insert  pattern  as  in  (a),  but  taken  with  our 
specially  arranged  specular  lighting  condition  plus  polarization 
image  processing.  This  is  the  IA  image  linearly  stretched  to  fit  an 
8  bit  display.  The  latent  fingerprints  are  revealed  and  at  the 
same  time  the  background  pattern  from  the  CD  insert  is  greatly 
suppressed.  The  upper  right  corner  of  images  (b)  and  (c)  appears 
brighter  because  those  areas  are  showing  the  specular  reflection 
image  of  the  light  source.  This  is  an  example  where  the  finger¬ 
print  stained  area,  rather  than  the  adjacent  clean  surface,  shows 
specular  reflection. 
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ues.  Figure  5(b)  presents  the  same  image  data  as  in  Fig. 
5(a),  but  enhanced  instead  by  histogram  equalization. 
This  enhancement  method  remaps  the  pixel  values  ac¬ 
cording  to  the  histogram  distribution  of  their  magnitudes, 
distributing  them  more  evenly  over  the  dynamic  range  of 


(c) 

Fig.  10.  (a)  Close-up  view  of  the  stainless-steel  Swiss  army 

knife  under  normal  viewing  conditions.  No  fingerprint  is  visible 
although  the  image  has  undergone  digital  linear  contrast  en¬ 
hancement.  (b)  The  same  area  of  the  stainless-steel  Swiss  army 
knife  as  in  (a)  but  taken  with  our  specially  arranged  specular 
lighting  condition  plus  polarization  processing.  The  latent  finger¬ 
prints  are  revealed  in  the  degree  of  polarization  image,  (c)  The 
same  area  of  the  stainless-steel  Swiss  army  knife  as  in  (a),  but 
taken  with  our  specially  arranged  specular  lighting  condition 
plus  polarization  image  processing.  This  is  the  IA  image.  The  la¬ 
tent  fingerprints  are  revealed  and  at  the  same  time  the  back¬ 
ground  pattern  from  the  book  cover  is  greatly  suppressed. 


(C) 

Fig.  11.  (a)  Transparent  tape  under  normal  viewing  conditions. 

No  fingerprint  is  visible.  The  tape  itself  is  barely  recognizable 
since  it  is  transparent,  (b)  Fingerprint  found  on  the  sticky  side  of 
the  tape  using  specular  reflection,  (c)  Fingerprint  found  on  the 
sticky  side  of  the  tape  using  polarization.  The  image  is  the  IA  im¬ 
age.  Note  that  no  ordinary  digital  contrast  enhancement  is  used 
on  any  of  the  images  in  this  figure. 

the  display  (see  Ref.  39).  These  two  images  illustrate  the 
latent  nature  of  the  fingerprint:  The  natural  contrast  is  so 
low  that  not  only  can  the  unaided  human  eye  not  detect 
it,  but  even  widely  used  digital  image  enhancements  do 
not  reveal  its  presence.  Figures  5(c)  and  5(d)  present  im¬ 
ages  taken  with  only  the  specular  illumination  turned  on 
and  digitally  enhanced  with  the  same  methods  as  used  to 
produce  Figs.  5(a)  and  5(b),  respectively.  Imaging  the 
light  specularly  reflected  from  the  surface  yields  a  major 
enhancement  not  achievable  with  the  digital  enhance¬ 
ments  alone,  an  enhancement  traditionally  achieved  with 
powders  and  chemicals,  but  completely  without  destruc¬ 
tive  side  effects.  Figures  5(e)  and  5(f)  are  zoomed-in  views 
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showing  the  detailed  quality  of  the  recovered  fingerprint 
images.  As  an  example  comparison  with  the  fingerprint 
quality  lifted  by  a  conventional  method,  Fig.  5(g)  shows 
the  same  fingerprint  being  lifted  after  being  dusted  with 
forensic  black  magnetic  powder.  Figure  5(h)  shows  the 
fingerprint  lifted  by  a  forensic  sticky  tape  after  dusting. 
The  fingerprint  image  quality  recovered  by  the  proposed 
noncontact  method  is  more  consistent  compared  with  the 
results  obtained  by  powdering.  It  is  difficult  to  spread  the 
powder  evenly  across  the  whole  fingerprint,  and  an  area 
with  too  much  or  too  little  powder  will  be  lifted  with  lower 
quality.  Furthermore,  the  process  of  lifting  by  sticky  tape 
can  itself  introduce  missing  parts  in  the  fingerprint. 

The  results  of  another  experiment  employing  our  non- 
invasive  optical  method  are  illustrated  in  Fig.  6.  In  this 
case,  the  surface  with  the  latent  fingerprint  was  the  paper 
cover  of  an  ordinary  desk  calendar  [Fig.  6(a)],  which  pre¬ 
sented  a  greater  challenge  than  the  solid  colored  metallic 
surface.  The  paper  surface  contained  a  printed  pattern 
whose  light  absorption  interferes  with  the  optical  detec¬ 
tion  of  the  fingerprint.  We  used  both  specular  reflection 
and  polarization  analysis  to  extract  the  latent  fingerprint. 
For  the  polarization  analysis,  images  were  taken  with  a 
linear  polarization  analyzer  mounted  in  front  of  the  cam¬ 
era  and  oriented  at  three  different  angles.  The  picture 
displayed  in  Fig.  6(b)  is  the  value  image  of  the  derived 
quantity  IA  [see  Eqs.  (2)].  We  emphasize  here  that  Fig. 
6(b)  is  not  an  image  of  the  ordinary  intensity  distribution, 
but  rather  a  mapping  of  a  certain  physical  quantity  de¬ 
rived  from  the  polarization  distribution  of  the  light  com¬ 
prising  the  image.  The  specular  component  of  the  surface 
reflection  is  now  evident,  and  the  background  pattern  is 
gone.  Figure  6(c)  shows  a  cropped  close-up  of  the  finger¬ 
print  area  of  the  image  seen  in  Fig.  6(b).  These  results 
lead  to  two  additional  conclusions:  First,  the  noninvasive 
optical  method  can  extract  latent  prints  from  some  paper 
surfaces  as  well  as  from  smoother  surfaces;  and  second, 
the  processing  of  the  polarization  information  in  the  im¬ 
age  can  further  enhance  the  quality  of  the  recovered  la¬ 
tent  fingerprint  under  certain  conditions. 

Sometimes  the  specular  reflection  component  can  be 
obscured  by  more  intense  diffuse  reflection.  We  applied 
our  methods  to  such  a  case,  deliberately  picking  one  of  the 
strongest  diffuse  reflectors,  a  white  cotton  lining  under¬ 
neath  a  soft  clear  plastic  CD  sleeve  (Fig.  7).  Figure  7(a) 
shows  an  image  taken  without  a  polarizing  filter  in  front 
of  the  camera.  The  diffuse  white  light  is  so  strong  that  the 
specular  reflection  output  can  barely  enhance  the  latent 
fingerprint  on  the  plastic  surface.  However,  the 
polarization-based  analysis  followed  by  the  histogram 
equalization  readily  enhances  the  latent  fingerprint  [Figs. 
7(b)  and  7(c)]. 

Figures  8-10  illustrate  the  fingerprint  detection  and 
lifting  capabilities  of  our  new  methods  applied  to  several 
common  items  and  surfaces.  Note  that  our  method  is  not 
restricted  to  viewing  the  surface  at  a  right  angle,  and  our 
use  of  polarization  is  not  restricted  to  glass  and  is  not 
used  for  removing  glare. 

The  sticky  side  of  a  tape  has  traditionally  caused 
trouble  with  invasive  methods,  especially  methods  that 
apply  powder  to  the  surface.  Most  powders  and  reagents 
can  stick  easily  anywhere  on  the  sticky  surface,  not  just 
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the  fingerprint  area.  In  contrast,  the  sticky  side  of  the 
tape  is  ideally  suited  to  our  method.  The  sticky  material 
is  a  thin  coat  of  pliable  semitransparent  dielectric  that 
fits  our  surface  model  perfectly.  Whether  the  latent  fin¬ 
gerprint  mark  is  formed  by  skin  residue  or  by  a  plastic 
mark  formed  on  the  sticky  surface,  our  surface  model  pre¬ 
dicts  that  high-contrast  intensity  or  polarization  images 
can  be  formed  with  proper  lighting.  We  tested  our  method 
on  a  piece  of  transparent  packing  tape  and  the  results  are 
very  good  (Fig.  11).  An  example  of  plastic  fingerprint 
mark  detection  using  our  method  is  shown  in  Fig.  12. 

Figure  13  shows  a  series  of  experiments  using  both  the 
specular-reflection-only-based  and  the  polarization-based 
method  detecting  a  fingerprint  on  the  metal  case  of  a 
pump  (the  same  pump  surface  as  appeared  in  Fig.  5). 
Here  the  observing  view  angle  of  the  camera  is  varied 
(with  a  corresponding  change  in  the  incident  light  direc¬ 
tion  to  maintain  the  specular  reflection  relationship). 
Three  angles,  30,  45,  and  75  deg,  are  chosen  to  represent 
near-frontal,  near-Brewster’s,  and  near-grazing  view 
angles.  It  is  clear  that  the  near-grazing  angle  is  unsuit¬ 
able  because  it  is  difficult  to  see  the  fingerprint  pattern. 
On  the  other  hand,  the  light  from  the  light  source  used 
here  is  definitely  unpolarized.  As  predicted  by  the  Fresnel 
equation,  near  the  Brewster  incidence  angle  of  the  sample 
surface,  the  reflected  light  from  the  clean  sample  surface 
has  a  large  degree  of  polarization  while  the  light  reflected 


(b) 


Fig.  12.  (a)  Piece  of  hardened  epoxy  resin  under  ordinary  light¬ 

ing  conditions  and  view.  No  fingerprint  is  visible  even  after  a  lin¬ 
ear  rescale  to  fit  an  8  bit  display,  (b)  Plastic  fingerprint  mark  re¬ 
vealed  on  the  same  piece  of  hardened  epoxy  resin  as  in  (a)  in  the 
degree  of  polarization  image.  No  ordinary  digital  contrast  en¬ 
hancement  is  used  on  this  image. 
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(e)  (0 

Fig.  13.  Comparison  of  fingerprint  lifting  at  different  view  angles  (varying  both  the  incident  angle  of  the  light  source  and  the  view  angle 
of  the  camera  simultaneously  to  maintain  the  specular  reflection  condition  described  in  our  theory).  All  images  are  for  the  same  finger¬ 
print  taken  within  —  1  h  of  the  experiment  session.  View  angles  (the  angle  between  the  surface  normal  of  the  sample  to  the  view  direction 
of  the  camera)  are  —30  deg  for  (a)  and  (b),  45  deg  for  (c)  and  (d),  and  75  deg  for  (e)  and  (f).  Images  (a),  (c),  and  (e)  are  contrast  enhanced 
(linear  gray-level  stretch)  U,  representing  the  specular-reflection-based  method.  Images  (b),  (d),  and  (f)  are  contrast  enhanced  (linear 
gray-level  stretch)  A,  representing  the  polarization-based  method.  It  is  clear  that  at  a  near-grazing  angle  (75  deg)  it  is  difficult  to  see  the 
fingerprint  pattern.  At  near-frontal  view  (30  deg)  the  polarization  signal  is  very  weak,  so  after  contrast  enhancement  the  result  is  very 
noisy.  We  get  the  best  results  at  —45  deg. 


from  the  fingerprint-tainted  area  is  still  relatively  unpo¬ 
larized.  This  creates  a  good  contrast  in  several  kinds  of 
polarization  images  when  the  incident  light  angle  is  near 
the  Brewster  incidence  angle  of  the  sample  surface  and 
relatively  poor  contrast  at  other  angles.  However,  at  ex¬ 
actly  normal  incidence  to  the  sample  surface  the  reflected 
polarization  from  the  clean  sample  surface  is  the  same  as 
the  incident  one,  unpolarized,  and  is  not  much  different 
from  the  light  reflected  from  the  fingerprint-deposited 
area.  Therefore  we  see  poor  results  in  all  polarization  im¬ 
ages  when  light  from  our  light  source  is  incident  at  an  ex¬ 


actly  normal-incidence  angle  to  the  sample  surface.  In 
summary  it  is  best  to  pick  a  viewing  angle  close  to  Brew¬ 
ster’s  angle,  typically  from  45  to  60  deg  from  the  surface 
normal. 

Figure  14  is  exactly  the  same  experiment  setup  as  in 
Fig.  13,  except  that  these  images  were  taken  under  an  al¬ 
most  darkroom  environment  (room  lights  turned  off, 
doors  and  windows  shut)  while  Fig.  13  was  taken  under 
no  special  control  of  ambient  room  light.  In  fact  we  would 
like  to  point  out  that  all  experiment  pictures  except  Fig. 
14  in  this  paper  were  taken  without  special  control  of  am- 
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bient  room  light  (i.e.,  room  light  from  the  ceiling  was  left 
on;  doors  and  windows  were  left  as  is).  We  note  that  our 
method  works  well  without  the  need  of  a  highly  controlled 
laboratory  darkroom.  Please  compare  corresponding  im¬ 
ages  in  Figs.  13  and  14.  One  can  hardly  see  any  signifi¬ 
cant  improvement  in  Fig.  14  where  experiments  were 
taken  in  a  controlled  darkroom  environment.  In  fact  they 
are  almost  identical.  The  reason  behind  this  result  can  be 
described  as  follows:  Irradiance  is  related  to  both  the 
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power  of  the  light  source  and  the  distance  between  the 
light  source  and  the  surface.  In  a  typical  office  or  resi¬ 
dence  room  the  lighting  is  not  much  stronger  than  a  flash¬ 
light  or  desk  lamp  that  would  be  used  in  our  setup.  Fur¬ 
thermore  the  light  used  to  detect  fingerprints  can  easily 
have  the  advantage  of  being  much  closer  to  the  surface 
under  examination.  In  our  experiment  we  put  a  lux  meter 
on  the  surface  and  we  got  —2000  lux  reading  when  both 
the  room  light  and  the  dedicated  light  source  were  on  and 
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Fig.  14.  Exact  same  fingerprint  target  and  the  same  experiment  as  shown  in  Fig.  13,  with  only  one  important  difference:  The  pictures 
shown  here  were  taken  with  all  ordinary  room  light  shut  off  and  the  door  shut  to  create  a  almost  darkroom  environment.  We  would  like 
to  point  out  that  all  experiment  pictures  shown  in  this  paper  except  pictures  in  this  figure  were  taken  without  any  particular  ambient 
light  control,  i.e.,  the  room  lights  from  the  ceiling  were  not  shut  off  and  doors  were  not  closed.  It  is  clear  that  our  method  performs  equally 
well  with  a  reasonable  amount  of  ambient  light  and  does  not  need  a  special  photography  darkroom.  The  corresponding  images  in  Fig.  13 
look  practically  identical.  We  used  a  flux  meter  to  measure  the  irradiance  difference  at  the  surface  of  the  fingerprint  sample  for  both 
ambient  light  on  and  ambient  light  off,  and  the  readings  are  around  2000  and  1900  lux,  respectively.  It  is  clear  that  as  long  as  our 
controlled  light  source  dominates  the  irradiance  at  the  surface  of  interest,  the  ambient  light  has  little  effect  on  the  effectiveness  of  our 
method  and  thus  we  can  safely  apply  our  method  directly  at  many  crime  scenes  without  the  need  to  bring  the  sample  to  a  dedicated 
laboratory  darkroom. 
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a  1900  lux  reading  when  we  turned  off  the  room  light  and 
shut  the  doors  and  windows  to  create  a  darkroom  envi¬ 
ronment  leaving  only  the  dedicated  light  source  illumi¬ 
nating  the  surface.  These  numbers  indicate  that  the  am¬ 
bient  light  accounted  for  only  —5%  total  irradiance  of  the 
surface,  and  the  dedicated  light  source  clearly  dominated. 
This  means  that  our  method  can  be  applied  directly  at 
many  crime  scenes  without  the  need  to  bring  samples 
back  to  a  special  laboratory  class  darkroom.  This  state¬ 
ment  should  not  be  misinterpreted  to  mean  that  we  claim 
that  our  method  works  in  all  possible  ambient  light  con¬ 
ditions  without  some  control.  It  is  well  known  that  the  ir¬ 
radiance  of  direct  sunlight  on  a  sunny  day  is  several  or¬ 
ders  of  magnitude  stronger  than  any  man-made  light 
source,  so  one  should  not  try  to  perform  this  method  un¬ 
der  direct  sunlight.  For  indoor  environments  it  may  gen¬ 
erally  not  be  a  problem  to  control  stray  light  by  either 
turning  off  some  lights  or  avoiding  direct  illumination  of  a 
room  light  or  temporarily  blocking  certain  ambient  light. 
If  a  particularly  strong  light  is  coming  from  a  particular 
direction,  one  may  simply  use  that  light  as  the  main  light 
source.  The  main  point  we  are  raising  here  is  that  our 
method  does  not  require  a  darkroom  as  much  as  does  de¬ 
veloping  a  traditional  camera  film,  where  any  stray  light 
would  expose  and  ruin  the  film.  Our  method  only  needs  to 
have  one  dominating  light  source  with  a  known  and  con¬ 
trollable  direction  of  illumination. 

4.  THEORY  OF  THE  APPLICATION  OF 
SPECULAR  REFLECTION  TO  LATENT 
FINGERPRINTS 

Modeling  surface  reflection  on  a  microscopic  scale  is  com¬ 
plicated  and  depends  heavily  on  the  detailed  knowledge  of 
the  molecular  material  composition  of  the  surface  mate¬ 
rial.  However,  macroscopically,  a  more  general  model  can 
be  used  that  applies  to  a  wider  range  of  surfaces  without 
the  need  for  details  about  the  surface  with  acceptable  re¬ 
duction  in  accuracy.  This  is  desirable  in  many  practical 
applications,  notably  in  computer  vision  and  graphics, 
where  the  details  of  the  chemical  and  physical  composi¬ 
tion  of  surfaces  are  not  known  or  are  not  of  vital  impor¬ 
tance.  Beginning  with  Refs.  40  and  41,  many  surface  re¬ 
flection  models  based  not  on  the  exact  chemical 
composition  but  rather  on  a  plausible  statistical  model  of 
the  surfaces  were  proposed  (see  also  Refs.  42  and  43).  A 
review  of  various  models  can  be  found  in  Refs.  44  and  45. 
Because  in  our  applications  we  intend  to  extract  the  fin¬ 
gerprint  without  using  any  chemical  analysis,  the  possi¬ 
bility  of  knowing  the  properties  of  the  surface  material 
beforehand  is  excluded.  However,  a  simple  model  that  de¬ 
scribes  a  general  trend  is  good  enough,  because  the  ulti¬ 
mate  form  required  for  a  fingerprint  image  is  that  of  bi¬ 
narized  black-and-white  regions  separating  the  ridge  and 
furrow  areas.  There  is  no  need  to  recover  or  to  predict  the 
exact  brightness  differences  in  the  gray-level  images 
taken  for  the  purpose  of  recovering  fingerprint  marks. 
The  simple  Phong  model43  and  Lambertian  model,40  both 
widely  used  in  many  computer  vision  and  graphics  algo¬ 
rithms,  satisfy  these  purposes. 

Macroscopically,  two  well-known  general  types  of  re¬ 
flection  can  be  named.  The  Lambertian  model  describes  a 
surface  producing  perfectly  diffuse  reflection  as 
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I  =  Ipkd  cos  6  =  Ipkd(n  •  Z) ,  (3) 

where  I  is  the  intensity  of  the  image  point  sensed  by  the 
camera,  Ip  is  the  point  light  source’s  intensity,  k ^  is  the 
reflection  coefficient  (either  for  a  particular  wavelength  or 
for  a  particular  camera’s  spectral  response),  and  6  is  the 
angle  between  the  surface  normal  h  and  the  unit  vector  l 
in  the  direction  of  the  light  source  as  viewed  from  the 
point  of  reflection  (see  Ref.  44).  Note  that  the  diffusely  re¬ 
flected  light  has  the  same  intensity  for  all  viewing  direc¬ 
tions. 

Another  type  of  reflection  is  that  of  highlights,  or  mir¬ 
rorlike  reflection  observed  on  many  smooth  surfaces.  A 
more  subdued  version  is  usually  called  sheen.  The  Phong 
model  is  given  by 

h  =  Ia\kaOdx  +faJp\[kdOd\  cos  0+  W{0) cos"  a],  (4) 

where  X  is  the  wavelength  of  the  light,  subscript  a  de¬ 
notes  the  ambient  light  source,  subscript  p  denotes  the 
point  light  source,  subscript  d  denotes  a  diffuse  reflection 
component,  the  new  symbol  O  denotes  color  components 
in  human  and  digital  color  vision  components,  fatt  is  the 
inverse  square  of  the  decay  distance  of  a  point  light 
source  intensity,  W(0)  is  the  diffuse  reflection  coefficient  of 
the  surface  with  a  point  light  source  angle  of  incidence  0, 
and  a  is  the  angle  between  the  exact  view  direction  pre¬ 
dicted  by  the  law  of  reflection  and  the  actual  view  direc¬ 
tion  [as  shown  in  Fig.  1(d)].  For  the  current  study,  the 
most  important  information  derived  from  Eq.  (4)  is  that 
the  intensity  is  proportional  to  cos71  a.  This  gives  a  simple 
way  to  model  a  rapid  decay  in  intensity  if  the  view  angle 
is  different  from  that  predicted  by  the  law  of  reflection. 
This  term  suggests  that  if  the  camera,  the  light  source, 
and  the  surface  being  inspected  are  arranged  in  a  way 
predicted  by  the  reflection  law  [see  Fig.  1(d)],  the  image 
point  for  the  original  smooth  surface  without  skin  residue 
will  show  an  intensity  typical  for  specular  reflection, 
while  the  area  with  skin  residue  will  have  much  less 
specular  reflection  due  to  the  slight  change  in  the  direc¬ 
tion  of  surface  normal  caused  by  the  skin  residue.  The 
more  mirrorlike  a  surface  can  be  modeled,  the  larger  the 
power  of  cosine  decay  it  exhibits,  which  means  better  con¬ 
trast  in  our  specular-reflection-based  latent  fingerprint 
detection  and  lifting  technique. 

Providing  that  all  other  factors  are  equal,  the  intensity 
of  the  specular  reflection  component  is  in  general  much 
stronger  than  the  diffuse  reflection  component.  Although 
this  statement  is  not  always  true,  it  has  been  widely  ac¬ 
cepted  as  a  good  rule  of  thumb  in  the  majority  of  practical 
situations.26  Since  specular  reflection  has  a  tendency  of 
concentrating  reflected  energy  in  a  small  solid  angle,  as 
opposed  to  the  diffuse  reflection  that  spreads  all  the  re¬ 
flected  energy  into  a  full  hemisphere,  the  same  amount  of 
reflected  energy  will  result  in  a  much  greater  flux  density 
in  specular  reflection  and  thus  in  image  brightness. 
Specular  reflection  tends  to  be  reflected  only  once  from 
the  smooth  surface,  while  diffuse  reflection  gives  light 
that  experiences  a  multiple  scattering  inside  the  surface 
before  reemerging.  Each  scattering  only  weakens  the  in¬ 
tensity  but  seldom  enhances  it.  Wolff26  experimentally 
measured  the  ratio  between  specular  and  diffuse  reflec- 
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tion  intensities  for  several  different  surfaces  and  reported 
ratios  varying  from  —150:1  to  250:1.  For  many  digital 
sensors  with  8  bit  brightness  resolution,  this  is  close  to 
the  maximum  intensity  ratio  of  255:1.  This  gives  a  strong 
support  to  our  main  assumption  that  the  specular  reflec¬ 
tion  component  is  in  general  stronger  than  the  diffuse 
one.  Our  experimental  results  so  far  also  support  the  va¬ 
lidity  of  such  an  assumption. 

In  Ref.  26  another  important  theoretical  result  relating 
to  a  fingerprint  detection  issue  was  reported,  i.e.,  if  the  re¬ 
flection  coefficient  of  the  diffuse  component  is  more  than 
—  1/33  the  reflection  coefficient  of  the  specular  compo¬ 
nent,  the  diffuse  component  can  be  stronger  or  at  least 
comparable  to  the  specular  component.  If  the  underlying 
surface  consists  of  complicated  patterns  similar  in 
strength  and  spatial  frequency  to  the  latent  fingerprint 
pattern  on  top  of  it,  the  method  based  on  purely  specular 
reflection  is  not  satisfactory.  This  is  the  point  where  the 
polarizer  should  be  used. 

The  behavior  of  the  specular  component  is  governed  by 
the  well-known  Fresnel  reflection  coefficients  formula46: 

E0r  \  rii  cos  0t  -  nt  cos  0t  sin(  6t  -  0t) 

E0i  J  ±  nt  cos  0t  +  nt  cos  0t  sin(  0t  +  0t)  ’ 

E0r  \  nt  cos  0i  -  nt  cos  0t  tan(  0t  -  0t) 

—  = - = - ,  (5) 

E0i )  %  Hi  cos  0t  +  nt  cos  0i  tan(  0t  +  0t) 

where  the  subscripts  i,  t,  and  r  stand  for  incident,  trans¬ 
mitted,  and  reflected  components,  respectively.  The  sub¬ 
scripts  _L  and  II  are  related  to  the  plane  of  incidence.  In 
Fig.  1(b)  the  plane  of  this  paper  is  the  plane  of  incidence. 
In  the  case  of  specular  reflection  it  contains  both  the  inci¬ 
dent  and  the  reflected  light-wave  vectors.  It  is  well  known 
(see,  e.g.,  Ref.  46)  that  ry  can  be  exactly  zero  at  Brewster’s 
angle  0g,  which  is  given  as 

tan  0B  =  nt/ni.  (6) 

If  light  is  incident  from  the  air,  —  1,  while  nt  varies  from 
1.4  to  2.0  for  most  of  dielectrics  in  the  visible  band  (wave¬ 
length  of  —400-700  nm).46  Equation  (6)  shows  that  the 
corresponding  Brewster’s  angles  vary  from  55  to  74  deg, 
respectively.  Although  there  are  certain  materials  with  a 
higher  refractive  index,46  we  confine  our  discussion  to  the 
above-mentioned  range  of  nt.  Therefore,  when  we  con¬ 
sider  the  angle  of  incidence  between  55  and  74  deg,  the 
reflected  light  is  highly  partially  polarized  with  the  plane 
of  polarization  perpendicular  to  the  plane  of  incidence. 
This  case  is  referred  to  as  horizontal  polarization  with  re¬ 
spect  to  the  surface  being  inspected.  So  far  we  have  only 
discussed  dielectric  surfaces  that  are  adequate  in  most 
cases  since  pure  metallic  surfaces  are  rather  rare  in  ev¬ 
eryday  life.  Pure  metal  surfaces  are  oxidized  quickly  and 
the  actual  layer  responsible  for  specular  reflection  is  often 
either  the  oxides  on  the  surface  or  the  protective  painting 
layer  that  is  also  a  dielectric  material.  In  fact,  a  lot  of 
metallic-looking  merchandise  today  is  actually  coated 
with  highly  reflective  dielectric  paints.  In  cases  when  the 
underlying  pure  metal  reflects  more  light  than  the  upper 
dielectric  coating,  the  proposed  method  may  not  work. 


5.  DISCUSSION 

The  currently  popular  latent  fingerprint  detection  and  ex¬ 
traction  methods  used  by  law  enforcement  agencies  in¬ 
clude,  but  are  not  limited  to,  powdering,  Sudan  Black 
staining,  iodine  fuming,  ninhydrin  (sometimes  followed 
by  further  enhancement  with  zinc  chloride)  and  1,8- 
diazafluoren-9-one  (DFO)  application,  silver  nitrate  de¬ 
velopment,  cyanoacrylate  (glue)  fuming,  Gentian  Violet 
staining,  vacuum  metal  deposition,  laser-excited  lumines¬ 
cence,  and  reflected  ultraviolet  imaging  system 
(RUVIS).1-4  While  this  list  may  seem  long,  there  is  still 
need  for  new  methods,  because  every  existing  method 
tends  to  be  unsuitable  for  some  surfaces,  due  either  to  its 
inadequacy  in  lifting  the  print  from  or  to  its  damaging 
side  effects  to  the  surfaces.  In  particular,  the  chemical 
and  physical  processing  directly  applied  to  the 
fingerprint-bearing  surface  to  extract  latent  fingerprints 
can  potentially  inflict  deleterious  effects  upon  the  finger¬ 
print,  the  operator,  and/or  the  object  surface  being 
examined. 1-3,10,18  For  example,  the  iodine  vapor  in  the  io¬ 
dine  fuming  method  is  highly  corrosive  and  toxic.2  Thus, 
in  practice,  often  valuable  and/or  irreplaceable  objects  are 
not  searched  for  fingerprints  at  all,2,3,7  except  in  a  few 
major  cases  involving  extremely  serious  crimes.  Further¬ 
more,  the  chemicals  used  to  enhance  fingerprint  contrast 
or  to  induce  luminescence  may  need  a  long  processing 
time,  are  sometimes  toxic,  environmentally  unfriendly,  or 
radioactive.  The  process  to  speed  up  the  chemical  reaction 
can  be  dangerous,  e.g.,  sodium  hydroxide  used  to  speed  up 
cyanoacrylate  fuming  can  generate  extreme  heat  if  the 
two  come  into  contact.4  Samples  are  often  baked  to  high 
temperature  after  many  fuming  procedures  to  speed  up 
the  print  development.  Chemicals  used  for  fingerprint  en¬ 
hancement  can  often  be  harmful  to  the  operator  if  not 
handled  correctly  using  proper  procedures  and  protective 
equipment  because  they  are  designed  to  react  with  or  ad¬ 
here  to  the  fingerprint  residues,  which  are  the  same  ma¬ 
terial  found  on  human  skin.3  Some  chemicals  require  spe¬ 
cific  solvents  that  have  undesirable  side  effects,  e.g.,  the 
phenol  in  the  solution  for  Gentian  Violet  is  highly  caustic 
and  poisonous.2 

Laser-induced  fluorescence  was  one  of  the  first  optical 
methods  for  lifting  latent  prints,  utilizing  induced  lumi¬ 
nescence  of  the  fingerprint  material.1-3,10  However,  there 
are  several  factors  limiting  its  use.  The  natural  fluores¬ 
cence  signal  (without  the  prior  application  of  strongly 
fluorescent  chemicals  or  powders)  is  in  general  very  weak. 
Thus  the  laser  used  must  operate  in  a  specific  frequency, 
and  must  have  a  high  enough  power  rating.  Such  a  laser 
device  is  fairly  expensive  and  bulky  (due  to  power  and 
cooling  requirements),  and  thus  it  can  be  deployed  only  to 
a  few  large  and  well-funded  crime  laboratories  and  is  not 
easily  made  portable  outside  a  dedicated  crime 
laboratory.18  A  less  powerful  laser  or  arc  lamp  with  a  filter 
can  be  used  as  a  substitute  but  will  be  less  effective.  As  a 
result,  these  alternative  systems  need  chemical  enhance¬ 
ments.  The  fact  that  many  commonly  encountered 
fingerprint-laden  objects  found  at  a  crime  scene  contain 
organic  substances  that  also  fluoresce  when  excited  by  a 
laser  often  cause  significant  background  noise.1,2,10,13,17-19 
Thus,  laser-excited  luminescence,  like  other  existing 
methods,  cannot  be  applied  to  certain  types  of  surfaces.  It 
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has  also  been  found  that  the  signal  strength  of  the  natu¬ 
ral  fluorescence  of  fingerprints  varies  greatly  from  person 
to  person  and  even  from  time  to  time  of  the  same 
person.10  As  a  result,  in  real  applications  laser-excited  lu¬ 
minescence  techniques  are  more  often  used  with  the  aid 
of  fluorescence-enhancing  chemicals  to  get  better  and 
more  consistent  results.1,2,13,17-19  However,  the  use  of 
chemical  fluorescence  enhancers  negates  the  noncontact 
advantage  and  the  combined  method  reverts  back  to  an 
invasive  method. 

RUVIS  was  pioneered  by  research  at  the  National  Po¬ 
lice  Agency  of  Japan  and  was  originally  targeted  for  en¬ 
hancing  cyanoacrylate-developed  latent  prints.  It  is  found 
that  the  cyanoacrylate-developed  latent  prints  that  are 
translucent  under  visible  light  become  opaque  under  UV. 
Latent  prints  deposited  in  sebaceous  matter  or  oily  resi¬ 
due  can  sometimes  be  detected  by  RUVIS  before  the  ap¬ 
plication  of  enhancing  materials.  The  oil  strongly  absorbs 
UV  and  shows  up  as  dark  patterns  under  UV  light  and 
detector.  RUVIS  can  sometimes  detect  fingerprints  up  to 
a  year  old  in  a  purely  noncontact  style.1  The  drawback  is 
the  need  of  a  relatively  expensive  specialized  UV  light 
source  and  sensors.  UV  light  can  be  harmful  to  both  hu¬ 
man  eyes  and  skin,  so  proper  precaution  and  protective 
gear  are  necessary.  Bramble  et  al.47  proposed  using  a  la¬ 
ser  in  the  UV  band  to  induce  luminescence  that  is  also  in 
the  UV  band.  Although  the  UV  laser  is  not  considered  in¬ 
vasive,  they  find  that  the  luminescence  property  of  the 
fingerprint  material  decreases  significantly  after  being 
exposed  to  a  UV  laser  for  an  extended  period  of  time.47 

The  episcopic  coaxial  illumination  method  proposed  by 
Pfister20  uses  the  intensity  difference  between  specular 
and  diffuse  reflection  to  enhance  the  visibility  of  latent 
fingerprints.  The  design  always  looks  at  the  surface  from 
a  right  angle.  The  advantage  is  that  the  picture  will  al¬ 
ways  appear  in  frontal  view  suitable  for  archiving.  How¬ 
ever,  with  the  advent  of  digital  image  processing,  an  ob¬ 
lique  view  can  be  easily  reprojected  back  to  the  frontal 
view,  so  it  is  not  that  crucial.  We  have  shown  that  by  vary¬ 
ing  the  observation  angles  we  can  get  a  better  contrast  in 
some  cases  [compare,  e.g.,  Figs.  12(a)  and  12(b)].  We  have 
also  shown  that  the  partially  polarized  nature  of  oblique 
specular  reflection  can  significantly  enhance  the  contrast 
of  latent  fingerprints  and  at  the  same  time  suppress  the 
interfering  background  pattern.  Viewing  the  surfaces  only 
at  a  right  angle  precludes  the  use  of  polarization  because 
the  specular  reflection  at  a  right  angle  is  not  partially  po¬ 
larized  by  the  surface. 

There  is  a  commercial  product  called  the  fingerprint 
camera  that  has  existed  for  a  long  time.4  Some  may  con¬ 
fuse  it  as  yet  another  noncontact  optical  method  for  the 
detection  of  latent  fingerprints.  However,  this  device  is  in 
fact  an  ordinary  camera  customized  to  record  nonexem¬ 
plar  fingerprints  that  have  been  enhanced  by  other  meth¬ 
ods  (e.g.,  by  an  application  of  chemicals)  or  for  patent  fin¬ 
gerprints  that  are  already  visible.  The  customization 
includes  dedicated  lighting  to  reduce  uneven  lighting, 
shadow,  and  glare;  a  fixed-object  distance  and  fixed  focus, 
aperture,  exposure  time,  etc.  These  preset  camera  set¬ 
tings  enable  law  enforcement  officers  who  may  not  be 
photographic  experts  to  be  able  to  take  good  fingerprint 
pictures  consistently  for  court  use.  Such  a  fingerprint 


camera  does  not  have  any  contrast  enhancement  or  detec¬ 
tion  capability  for  hidden  latent  fingerprints. 

The  novel  optical  method  we  propose  here  is  capable  of 
recovering  high-quality  digital  images  of  hidden  latent 
fingerprints  without  the  application  of  any  chemicals  or 
physical  contact  with  the  examined  object.  Like  any  other 
existing  latent  fingerprint  enhancement  technology,  our 
method  has  its  limitations.  Our  method  is  designed  to 
take  advantage  of  the  intensity  and  polarization  differ¬ 
ences  between  specular  and  diffuse  reflections,  so  it  works 
best  on  a  relatively  smooth  and  nonporous  surface.  The 
chance  of  detecting  fingerprints  on  a  highly  porous  and 
absorbing  surface  like  certain  kinds  of  paper  using  our 
method  is  fairly  low.  However,  our  method  possesses 
unique  advantages  over  existing  methods  on  smooth  sur¬ 
faces  in  that  complex  patterns  on  the  surface  can  be  easily 
suppressed  optically  using  our  polarization  method.  Note 
that  while  the  most  common  use  of  a  photographic  polar¬ 
ization  filter  is  to  remove  glare,  our  use  of  the  polarizer  is 
quite  the  opposite.  Our  polarization  method,  in  a  sense, 
extracts  only  the  specular  glare  that  was  usually  treated 
as  a  mere  nuisance.  Specifically  we  discovered  that  inside 
the  specular  glare  under  certain  conditions  a  high- 
contrast  clean  latent  fingerprint  image  is  present. 

Menzel22  briefly  mentioned  the  possibility  of  using  po¬ 
larization  to  detect  fingerprints  on  glass.  We  have  shown 
that  polarization  can  be  used  to  enhance  fingerprint  vis¬ 
ibility  on  a  wide  variety  of  surfaces,  not  just  on  glass.  We 
also  show  that  polarization  can  be  used  to  remove  inter¬ 
fering  background  patterns  that  have  not  been  mentioned 
before.  Another  advantage  of  our  method  is  that  it  is  less 
affected  by  the  degradation  of  latent  fingerprints  over 
time  compared  with  methods  that  rely  on  chemical  reac¬ 
tions  with  the  organic  residues  or  water  in  latent  finger¬ 
print  marks.  Over  time  an  exposed  latent  fingerprint 
mark  will  lose  its  water  contents  via  evaporation,  and  the 
amino  acid  components  will  degrade  and  its  chemical 
properties  will  change.1-4  Since  our  method  does  not  rely 
on  water  or  detailed  chemical  composition  of  the  finger¬ 
print  mark,  our  method  is  less  likely  to  be  affected  by 
chemical  decomposition.  As  long  as  the  geometric  differ¬ 
ence  in  the  surface  remains,  our  method  can  be  used  ef¬ 
fectively.  The  example  pictures  shown  were  taken  up  to 
several  weeks  after  the  fingerprint  was  made  (and  kept  in 
our  labor atory/office  environment  undisturbed)  and  they 
show  little  degradation  of  quality. 

We  have  shown  that  our  method  can  work  under  regu¬ 
lar  room  light  without  the  need  of  a  highly  controlled 
laboratory  darkroom.  This  may  enable  law  enforcement 
officers  to  use  our  method  right  at  the  crime  scene  with¬ 
out  the  delay  of  sending  samples  back  to  a  special  crime 
laboratory.  This  effectively  means  that  our  method  can 
usually  examine  the  most  fresh  fingerprint  samples.  The 
only  requirement  is  a  strong  dominating  light  source  illu¬ 
minating  the  surface  examined  from  a  known  direction. 
This  can  be  easily  achieved  by  either  using  a  strong  por¬ 
table  light  source  close  to  the  target  surface  or  by  simply 
using  the  strongest  directional  ambient  light  source  found 
at  the  crime  scene.  For  most  indoor  crime  scenes,  the 
standard  fingerprint  search  procedure  calls  for  restricting 
the  access  to  the  crime  scene  with  only  the  fingerprint 
specialists  working  at  the  scene  during  the  fingerprint 
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search.  The  specialists  have  total  control  over  the  scene 
during  the  search  so  it  is  generally  not  a  problem  to  turn 
on  and  off  any  room  light.  It  is  also  quite  easy  to  use 
something  like  a  shade  to  temporarily  block  some  ambi¬ 
ent  light,  e.g.,  a  piece  of  cloth,  an  umbrella,  or  even  the 
body  of  the  specialist  because  the  size  of  a  fingerprint  is 
very  small.  In  other  words,  our  method  can  be  performed 
directly  in  most  crime  scenes  and  even  more  cases  with 
simple  common  sense  regarding  light  control.  There  is  es¬ 
sentially  no  need  for  a  special  darkroom  similar  to  a  film 
developing  darkroom. 

Because  of  the  great  variety  of  latent  fingerprints  that 
can  potentially  be  found  in  very  different  crime  scenes,  no 
single  latent  fingerprint  enhancement  method  can  handle 
all  possible  cases.  Since  every  method  targets  different 
physical  and  chemical  properties  of  different  kinds  of  fin¬ 
gerprints,  it  is  often  found  that  applying  more  than  one 
method  on  the  same  surface  will  unveil  different  finger¬ 
prints  for  different  methods.18  Thus  the  introduction  of  a 
new  latent  fingerprint  detection  method  that  uses  differ¬ 
ent  physical  principles  than  those  used  by  existing  meth¬ 
ods  and  one  that  will  not  interfere  with  other  methods 
should  be  beneficial  to  law  enforcement  efforts. 


6.  CONCLUSION 

We  have  introduced  a  novel  optical  method  to  detect,  en¬ 
hance,  and  lift  latent  fingerprints  that  are  otherwise  dif¬ 
ficult  to  see.  The  method  is  noninvasive  and  so  will  not  in¬ 
terfere  with  other  forensic  examinations  and  will  not 
inflict  deleterious  side  effects  on  the  surface.  The  equip¬ 
ment  required  is  much  less  expensive  than  most  other 
noncontact  methods  proposed  so  far.  The  new  method  is 
not  applicable  to  highly  porous  and  absorbing  surfaces 
but  works  well  on  most  other  surfaces.  The  new  method 
also  works  on  the  sticky  side  of  tape,  which  is  a  particu¬ 
larly  problematic  surface  for  many  invasive  methods.  The 
new  method  can  be  easily  performed  right  at  the  crime 
scene  without  the  need  to  bring  samples  to  a  specialized 
laboratory.  The  recovered  latent  fingerprint  image  has 
good  quality  compared  with  other  existing  methods.  Since 
it  does  not  interfere  with  other  methods,  it  is  advisable  to 
try  this  method  before  other  invasive  or  destructive  meth¬ 
ods. 
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Abstract:  A  thin  film  polarization  filter  has  been  patterned  and  etched 
using  reactive  ion  etching  (RIE)  in  order  to  create  8  by  8  microns  square 
periodic  structures.  The  micropolarization  filters  retain  the  original 
extinction  ratios  of  the  unpatterned  thin  film.  The  measured  extinction  ratios 
on  the  micropolarization  filters  are  -1000  in  the  blue  and  green  visible 
spectrum  and  -100  in  the  red  spectrum.  Various  gas  combinations  for  RIE 
have  been  explored  in  order  to  determine  the  right  concentration  mix  of  CF4 
and  02  that  gives  optimum  etching  rate,  in  terms  of  speed  and  under¬ 
etching.  Theoretical  explanation  for  the  optimum  etching  rate  has  also  been 
presented.  In  addition,  anisotropic  etching  with  1pm  under  cutting  of  a 
10pm  thick  film  has  been  achieved.  Experimental  results  for  the  patterned 
structures  under  polarized  light  are  presented.  The  array  of  micropolarizers 
will  be  deposited  on  top  of  a  custom  made  CMOS  imaging  sensor  in  order 
to  compute  the  first  three  Stokes  parameters  in  real  time. 
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1.  Introduction 

Polarization  vision  contains  important  information  about  the  imaged  environment,  such  as 
surface  shapes,  curvature  and  material  properties  ,  which  are  ignored  with  traditional  imaging 
systems  [1].  Polarization  vision  techniques  have  been  extensively  used  in  applications  such  as 
pupil-plane  speckle  imaging  [2],  passive  target  detection  [3],  underwater  imaging  [4]  and 
others.  Several  species  of  invertebrate,  such  as  cuttlefish,  mantis  shrimp  [5],  desert  ants  and 
various  other  species  of  insects  [6],  rely  on  contrast  enhancement  using  polarized  vision, 
which  is  a  vital  survival  mechanism  in  optically  scattering  media.  The  human  eye  perceives 
visual  information  in  terms  of  color  and  intensity  but  it  is  essentially  blind  to  polarization. 
Here,  we  are  developing  an  imaging  system  capable  of  extracting  almost  the  complete  set  of 
polarimetric  properties  for  partially  polarized  light  in  parallel  and  real  time.  This  sensory 
system  integrates  an  array  of  imaging  elements,  a  micropolarization  array  and  analog 
processing  circuitry  for  polarimetric  computation  at  the  focal  plane  in  order  to  achieve  a 
compact,  low  power  polarization  sensitive  system. 

Biologically  inspired,  polarization  difference  (contrast)  imaging  (PDI)  sensors  have  been 
one  of  the  dominant  research  topics  in  developing  polarization  sensitive  systems  [7-10]. 
Several  imaging  systems  have  been  made  where  polarization-contrast  information  has  been 
computed  at  the  focal  plane  [7,8]  or  with  bulky,  power  hungry  set-ups  composed  of 
CMOS/CCD  imaging  sensors,  electro/mechanically  controlled  polarization  filters  and 
DSPs/CPUs  [9-11].  These  systems  sample  the  environment  via  two  orthogonal  polarization 
filters  and  compute  contrast  information  at  the  focal  plane  with  translinear  circuits  or  in  the 
digital  domain  with  a  DSP/CPU.  The  sampling  of  the  environment  is  achieved  with  spatially 
distributed  polarization  orthogonal  filters  over  the  neighborhood  of  two  pixels  or  by 
temporally  sampling  the  scene  with  two  sequential  orthogonal  filters.  Tradeoffs  between  these 
two  approaches  are  reduction  of  spatial  resolution  in  the  former  vs.  reduction  of  frame  rate  in 
the  latter  system. 

Incorporating  pixel-pitch-matched  polarization  filters  at  the  focal  plane  has  been  first 
reported  by  Andreou  et  al.  [7,12,13],  where  birefringent  materials  and  thin  film  polarizers 
have  been  explored.  The  pixel  pitch  for  the  birefringent  and  thin  film  micropolarizer  arrays 
was  50pm  and  25pm,  respectively.  The  large  pixel  pitch  was  limited  by  the  wet  isotropic 
etching  technique  employed  for  patterning  the  thin-film  polarization  filter  and  by  the  in-pixel 
circuit  processing  for  extracting  polarization  contrast  information  in  the  birefringence  imaging 
system.  A  birefringent  micropolarization  element  array  with  pixel-pitch  size  of  25  pm  and 
128pm  was  reported  in  [14]  and  [8]  respectively.  The  large  pixel  pitch  in  these  polarimetric 
imaging  systems  limits  the  fidelity  of  the  imaged  environment,  which  is  a  major  shortcoming 
for  high  resolution  imaging  applications. 
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Another  avenue  of  research  has  been  the  design  of  micropolarization  array  for  extracting 
the  complete  set  of  polarimetric  properties  of  partially  linearly  polarized  light.  A 
micropolarization  array  with  three  spatially  distributed  polarizers  was  fabricated  and 
described  by  Guo  et  al.  [15].  The  polymer  thin  film  was  used  to  create  a  three  axial 
micropolarizer  array  with  25  micron  pixel  pitch  elements.  A  dual  axel  dichroic-on-threated- 
glass  micropolarizer  array  with  5pm  pixel  pitch  and  extinction  ratio  of  -330  was  reported  in 
[16].  Wire  grid  micropolarization  array  for  near  infrared  spectrum  have  been  reported  in  [17]. 

The  previously  reported  PDI  systems,  which  have  been  directly  inspired  from  biological 
systems,  compute  polarimetric  information  in  simplified  and  compact  form  [7,8,10].  In 
contrast,  the  complete  polarimetric  information  tends  to  be  far  more  complex  and  its 
computational  demands  usually  prevent  real  time  extraction  [10].  These  complex  polarization 
properties  are  fully  described  by  the  four  fundamental  parameters  known  as  the  Stokes 
parameters.  For  natural  lights,  which  are  usually  polychromatic  and  partially  polarized  and  for 
which  the  phase  information  between  the  components  is  not  available,  only  the  first  three 
Stokes  parameters  are  needed.  In  order  to  fully  determine  these  three  Stokes  parameters,  the 
imaged  scene  must  be  sampled  at  least  with  three  different  polarization  filters. 

We  are  developing  a  high  resolution  imaging  system  capable  of  extracting  the  first  three 
of  the  four  Stokes  parameters  from  the  imaged  environment  in  parallel  and  in  real  time.  This 
novel  sensory  system  integrates  imaging  array  with  10pm  pixel  pitch  with  a  photodiode  area 
of  8x8  pm  square,  pitch-matched  micropolarization  array  and  polarization  processing  at  the 
focal  plane. 

In  this  paper  we  solely  focus  on  the  microfabrication  steps  necessary  for  patterning  a 
commercially  available  thin  film  polarizer  in  order  to  create  a  dual  axel  micropolarization 
array  with  8pm  by  8pm  square  filters.  The  paper  is  organized  as  follows.  In  Section  2,  a 
theoretical  background  on  light  polarization  is  presented  in  order  to  introduce  a  technique  for 
polarimetric  computation  at  the  focal  plane.  Section  3  presents  the  micro  fabrication  steps 
necessary  to  manipulate  thin  film  polarizer.  RIE  effects  on  the  polarizer  are  theoretically 
analyzed  in  Section  4.  Experimental  data  is  presented  in  Section  5  and  concluding  remarks  are 
summarized  in  Section  6. 


2.  Overview  of  polarization  information  and  Stokes  parameters 

Stokes  parameters  can  describe  the  polarization  information  of  light.  There  are  different  ways 
to  express  Stokes  parameters,  one  of  which  is  presented  by  Eqs.  (1)  through  (4),  which 
describe  fully  the  polarization  state  of  the  electric-field  vector  [16]. 


So=I. 

St=  2/(0°,  0)-/ 


(1) 

(2) 


S2  =  2/(45°,  0)-/ 

S3  =  /  -2/(45°,#/ 2)  (4) 

In  Eqs.  (1)  through  (4),  It  is  the  total  intensity;  1(0°, 0)  is  the  intensity  of  the  e- vector 
filtered  with  a  0  degree  polarizer  and  no  phase  compensation  between  the  x  and  y 
components;  1(45° ,0)  is  the  intensity  of  the  e-vector  filtered  with  a  45  degree  polarizer  and  no 
phase  compensation  as  above;  and  I(45°f7t/2)  is  the  intensity  of  the  e- vector  filtered  with  a  45 
degree  polarizer  and  7t/2  phase  compensator  between  the  x  and  y  components. 

It  is  worth  mentioning  that  the  first  three  Stokes  parameters  describe  the  polarization  of 
light,  when  the  phase  information  between  the  components  is  not  available.  Therefore,  in 
order  to  describe  the  polarization  state  of  such  light  in  nature,  three  linearly  polarized 
projections  (for  example,  0  degree,  60  degree  and  120  degree  polarization,  or  0  degree,  45 
degree  and  90  degree)  or  two  (non-orthogonal)  linearly  polarized  projections  in  combination 
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Fig.  1.  Block  diagram  of  the  complete  imaging  system:  imaging  array,  processing  circuitry 
and  micropolarization  array 


with  the  total  intensity  are  needed.  The  latter  method  is  preferred  for  focal  plane 
implementation  since  it  only  requires  two  thin  film  polarizers  offset  by  45  degrees,  patterned 
and  placed  on  neighboring  pixels.  The  overall  thickness  of  the  complete  filter  will  be  thinner 
for  a  two-tier  vs.  a  three  tier  filter,  which  has  two  main  advantages.  The  first  advantage  is  in 
minimizing  light  attenuation  through  multiple  layers  and  increasing  the  angle  of  incidence. 
The  second  advantage  is  in  reduction  of  fabrication  steps  and  minimization  of  alignment 
errors 

The  patterning  of  the  thin  film  polarizer  is  similar  to  the  Bayer  pattern  used  in  color 
imaging  and  it  is  shown  in  Fig.  1  together  with  the  image  sensor.  The  image  sensor  is 
composed  of  an  array  of  256  by  256  photo  pixels,  and  the  noise  suppression  and  analog 
computation  circuitry  is  included  at  the  focal  plane.  The  micropolarizer  array  is  separately 
attached  on  the  image  sensor  with  the  pattern  shown  in  Fig.  1. 

In  the  image  sensor  presented  in  Fig.  1 ,  a  neighborhood  of  2  by  2  pixels  is  addressed  and 
accessed  simultaneously.  In  the  pixels  neighborhood  of  interest,  one  pixel  records  the  0 
degree  projected  polarized  image  (1(0°, 0)),  another  records  the  45  degree  projected  polarized 
image  {1(45°, 0))  and  two  pixels  record  the  unfiltered  intensity  image  (4).  The  polarimetric 
parameters  are  estimated  by  reading  out  all  four  pixels  in  parallel  [18]  and  scaling  them 
individually  at  the  periphery,  i.e.  away  from  the  imaging  array,  with  programmable  analog 
scaling  circuitry  in  accordance  to  the  first  three  Stokes  parameter  equations  (Eqs.  (1)  through 
(3)).  The  details  of  the  addressing  scheme  of  block  of  2  by  2  pixels  and  analog  processing 
circuitry  are  described  in  [18,19]. 

3.  Micro  fabrication  steps  for  thin  film  polarizer  manipulation 

A  commercially  available  thin  film  polarizer  is  used  to  create  an  array  of  micropolarizers.  The 
thin  film  polarizer  consists  of  an  iodine-doped  Polyvinyl  Alcohol  (PVA)  layer,  which  acts  as  a 
dense  array  of  thin  microscopic  wires.  These  microscopic  wires  are  formed  by  mechanically 
stretching  the  polymer  film,  allowing  the  molecules  of  the  PVA  to  align  in  the  direction  of 
stretching.  In  order  to  have  an  effective  polarizer,  the  size  and  spacing  between  the  thin 
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microscopic  wires  should  be  ~l/100th  and  ~l/10th  of  the  light’s  wavelength,  respectively  [20]. 
For  example,  for  blue  light  wavelengths  or  450nm  wavelength,  the  distance  between  the 
microscopic  wires  should  be  on  the  order  of  45nm,  while  the  thickness  of  the  wires  should  be 
around  5nm.  The  mechanical  stretching  of  the  polymer  creates  a  very  good  ~10pm  thick 
polarization  filter  for  the  visible  spectrum,  with  extinction  ratios  of  about  1000  in  the  blue  and 
green  spectrum  and  100  in  the  red  spectrum.  The  PVA  thin  film  is  placed  between  two  300pm 
thick  transparent  Cellulose  Acetate  Butyrate  (CAB)  or  polyethylene  layers  (Fig.  2-a).  The 
CAB  layers  provide  structural  stability  to  the  fragile  and  thin  PVA  layer. 

In  order  to  be  able  to  manipulate  the  PVA  layer,  at  least  one  of  the  protective  CAB  layers 
must  be  removed.  Since  CAB  is  a  form  of  acetate,  it  is  therefore  acetone  soluble.  Hence,  the 
sample  is  submerged  in  an  acetone  bath  for  30  minutes.  One  side  of  the  sample  is  taped  with 
an  acetone  resistant  tape,  i.e.  rubber  tape,  to  a  glass  substrate  in  order  to  protect  the  bottom 
layer  of  CAB  which  provides  structural  stability  for  the  PVA  layer.  The  sample  is  rinsed  with 
de-ionized  water  (DI)  in  order  to  remove  the  soften  CAB  layer.  The  DI  water  also  solidifies 
the  CAB  layer,  upon  which  the  sample  is  submerged  in  the  acetone  bath  again.  These  two 
steps  are  repeated  several  times,  until  no  residual  CAB  layer  remains  on  top  of  the  PVA  layer 
(Fig.  2-b). 

The  next  step  is  to  glue  the  PVA  to  a  substrate.  In  the  final  imaging  system,  the  PVA  will 
be  glued  directly  to  the  die  of  the  imaging  system.  For  testing  purposes,  however,  the  PVA  is 
glued  to  a  microscopic  glass  in  order  to  be  able  to  back  illuminate  the  filter  and  to  fully 
characterize  the  optical  properties  of  the  micropolarizer  elements.  A  UV  sensitive  adhesive 
promoter,  Dymax  AD420  [21],  is  used  between  the  PVA  and  the  IC/Glass  substrate  (Fig.  2-c). 
The  UV  adhesion  promoter  has  95%  transparency,  which  allows  for  minimal  absorption  of 
the  impingent  light  wave.  The  adhesive  promoter  exhibits  minimum  expansion  when  heated 
and  it  has  an  important  property  for  preserving  the  extinction  properties  of  the  PVA  layer. 

The  next  step  is  to  remove  the  bottom  CAB  layer  in  order  to  allow  patterning  the  PVA 
film.  This  step  is  similar  to  the  initial  step,  where  a  repeated  acetone  bath  and  DI  water  rinsing 
are  used.  The  final  structure  after  this  process  is  shown  in  Fig.  2-d. 

The  remaining  steps  describe  the  patterning  and  etching  process  of  the  PVA.  These  steps 
are  as  follow: 
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Fig.  2.  Microfabrication  steps  for  creating  a  dual  layer  polarization  structures 
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(1)  Heat  the  sample  to  a  temperature  of  95°C  for  5  minutes.  This  ensures  that  the  surface 
of  the  PVA  is  completely  dry.  An  adhesion  promoter,  Omnicoat  [22],  is  applied  directly  and 
spin  coated  at  3000  rpm  for  60  seconds.  The  sample  is  then  baked  for  1  minute  at  95°C.  The 
Omnicoat  layer  promotes  adhesion  between  the  PVA  and  the  SU-8  photoresist  (Fig.  2-e). 

(2)  Immediately  apply  an  SU-8  2015  negative  photoresist  [22]  on  top  of  the  PVA.  The 
negative  photoresist  is  hydrophobic  and  requires  the  surface  of  the  PVA  to  be  absolutely  free 
of  any  water  molecules.  If  the  sample  is  cooled  down  to  room  temperature,  water  molecules 
due  to  humidity  will  coat  the  surface  of  the  PVA.  The  adhesion  of  the  SU-8  will  be  virtually 
non  existent  when  patterning  10pm  or  smaller  structures.  Larger  structures  tend  not  to  have 
problems  with  adhesion  when  the  PVA  is  cooled  down  to  room  temperature. 

(3)  Spin  coat  the  photoresist  at  500rpm  for  10  seconds  and  then  at  3000  rpm  for  50 
seconds  with  500  rpm  per  second  acceleration.  The  resulting  photoresist  thickness  is  3pm. 

(4)  Bake  the  sample  at  65°  C  for  lmin  and  then  at  95°  C  for  2min.  It  is  recommended  that 
the  sample  cools  down  at  65°  C  for  1  min  in  order  to  gradually  decrease  the  temperature  of 
the  sample.  Gradual  increase  and  decrease  of  the  temperature  during  the  baking  process 
avoids  rapid  temperature  differences  and  prevents  the  photoresist  from  cracking  (Fig.  2-f). 

(5)  Expose  the  photoresist  at  375nm  wavelength  for  20  seconds  at  8mW/cnr  intensity. 
The  mask  used  to  pattern  features  for  the  imaging  sensor  contains  10  pm  by  10  pm  square 
patterns  (Fig.  2-g).  This  mask  allowed  us  to  closely  evaluate  isotropic  etching  properties  using 
an  electron  scanning  microscope. 

(6)  Post-bake  the  sample  at  65°  C  for  1  min  and  then  at  95°  C  for  2  min.  The  sample  is 
cooled  down  at  65°  C  for  1  min  in  order  to  gradually  decrease  the  temperature  and  minimize 
stress  and  cracking  on  the  photoresist. 

(7)  Develop  the  photoresist  for  4  min  in  an  SU-8  developer  and  gently  rinse  it  with 
isopropyl  alcohol.  If  white  colored  liquid  appears  on  the  surface,  the  photoresist  is  not 
completely  developed  and  it  is  submerged  in  the  developer  again  (Fig.  2-h). 

(8)  Selective  reactive  ion  etching  is  performed  on  the  sample.  A  mixture  of  gasses 
composed  of  28  seem  Ar,  30  seem  02  and  10  seem  CF4  is  used.  The  RF  power  is  150W,  at 
17°C  temperature  and  20mbar  pressure.  The  selected  ratio  of  the  appropriate  gasses  is 
optimized  to  maximize  the  etching  of  the  PVA  while  minimizing  the  destructive  etching  of 
the  SU-8  photo  resist.  The  etching  rate  of  the  PVA  is  0.5pm/min,  while  the  etching  rate  of  the 
SU-8  is  0.75pm/min.  Since  the  thickness  of  the  PVA  is  ~10pm,  the  sample  is  etched  for  ~20 
minutes.  The  thickness  of  the  SU-8  is  optimized  to  be  15pm  and  it  will  be  etched  completely 
by  the  time  the  PVA  layer  is  etched.  This  optimization  simplifies  the  microfabrication 
procedure  at  the  cost  of  slightly  damaging  the  PVA  structures  which  might  occur  due  to 
variations  in  the  PVA  and  SU-8  thickness. 

A  more  conservative  approach  can  be  employed,  where  the  SU-8  thickness  can  be 
increased  above  15pm.  Once  the  etching  of  the  PVA  layer  is  completed,  the  remaining 
photoresist  can  be  removed  with  an  SU-8  photoresist  striper.  The  Omnicoat  layer  helps  lift  the 
SU-8  photoresist  and  remove  it  from  the  PVA  structures.  The  RIE  provided  anisotropic 
etching  with  ~lpm  under  cutting,  compared  to  4pm  and  10pm  under  cutting  with  oxygen 
plasma  and  wet  etching,  respectively  (Fig.  2-i). 

The  next  step  is  to  add  a  second  layer  of  PVA  on  top  of  the  first  layer  offset  by  45  degrees 
via  a  UV  adhesive  promoter  and  etch  it  with  the  desired  mask  pattern.  First,  the  adhesive 
promoter  is  spin  coated  at  1500rpm  for  60  seconds  ensuring  ~lpm  epoxy  layer  on  top  of  the 
first  PVA  layer.  The  second  PVA  layer  is  placed  under  an  angle  starting  from  one  end  of  the 
sample  and  it  is  gradually  lowered  toward  the  other  side  of  the  sample.  This  step  helps  in 
eliminating  air  bubble  being  trapped  in  the  epoxy  layer  as  well  as  planarization  is  achieved 
due  to  the  liquid  state  of  the  epoxy  layer. 

As  an  alternative  method  of  merging  the  two  PVA  layers,  we  placed  the  adhesive 
promoter  on  top  of  the  first  PVA  layer  without  being  spin  coated.  The  second  PVA  layer  is 
gradually  lowered  toward  the  first  layer  while  maintaining  parallel  alignment  of  the  two 
layers.  Once  an  initial  contact  is  established  between  the  adhesive  promoter  and  the  second 
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PVA  layer,  due  to  the  dispersion  forces  of  the  adhesive  promoter,  the  adhesive  promoter 
flows  to  the  periphery  of  the  sample  without  any  air  bubbles  being  trapped.  The  excess 
adhesive  promoter  will  flow  outside  the  sample  as  the  top  layer  is  lowered  toward  the  first 
layer.  When  the  second  PVA  layer  is  completely  pressed  against  the  first  layer,  the  sample  is 
exposed  with  UV  light.  This  procedure  was  performed  using  an  aligner,  where  the  control  of 
the  contact  pressure  can  be  used  to  vary  the  thickness  of  the  adhesive  promoter  layer.  In  both 
cases  reliable  repeatability  of  merging  the  two  PVA  layers  was  achieved  with  adhesive  layer 
of  ~  1  pm. 

Following  the  adhesion  procedure,  the  top  CAB  layer  is  removed  with  acetone  in  order  to 
be  able  to  pattern  the  PVA  layer.  This  step  is  similar  to  the  initial  step,  where  a  repeated 
acetone  bath  and  DI  water  rinse  is  used.  The  final  structure  after  this  process  is  shown  in  Fig. 
2-j.  The  remaining  steps  for  patterning  the  top  layer  of  PVA  are  identical  to  the  one  for 
patterning  the  bottom  layers.  These  steps  are  shown  in  Fig.  2-k  through  Fig.  2-n.  The  final 
two-axial  micropolarization  filter  is  shown  Fig.  2-n,  with  a  total  thickness  of  around  20pm. 

Alignment  structures  were  used  on  both  the  PVA  layer  and  the  CMOS  imager,  in  order  to 
properly  align  the  various  layers.  The  first  layer  of  PVA  was  aligned  on  the  CMOS  image 
sensor  using  four  alignment  makers  on  the  corners  of  the  image  sensor.  The  second  PVA 
layer  was  aligned  with  the  first  PVA  layer  using  different  sets  of  alignment  makers  placed 
throughout  the  sample  in  order  to  increase  precision. 

4.  Reactive  Ion  Etching  (RIE)  Effects  on  the  PVA 

Cold  plasma  or  RIE  has  been  widely  used  to  modify  or  etch  polymers.  The  etching  process  of 
polymers  is  linearly  dependent  on  the  concentration  of  the  atomic-oxygen  free  radicals  [23]  or 
on  the  number  of  oxygen  atoms  consumed  during  the  etching  process  [24] .  Since  the  PVA  is 
an  unsaturated  polymer,  the  etching  process  is  represented  as  addition  to  unsaturated  moieties 
[24,25].  The  addition  of  oxygen  to  the  unsaturated  PVA  creates  a  saturated  radical  with  a 
weakened  C-C  bond.  Any  subsequent  attack  by  free  oxygen  radicals  will  break  the  C-C  bond 
and  it  will  divide  the  saturated  molecule.  Carbon  monoxide  (CO)  and  carbon  dioxide  (C02) 
are  released  during  the  etching  procedure  (equation  5). 

0-0  (5) 

I 

-C=C-  +  02:  ->  -C-C-  +02->  -C=0  +CO 

In  order  to  increase  the  etching  rate  of  the  PVA,  the  concentration  of  oxygen  atoms  must 
be  increased.  This  is  achieved  with  the  addition  of  fluorine  gases,  such  as  CF4,  CF3,  C2F6,  SF6 
and  others.  In  our  experiment,  CF4  was  used.  The  enhanced  etching  rate  is  due  to  the 
increased  density  of  electrons,  as  well  as  increased  energy  of  electrons  in  the  RIE.  At  the 
molecular  level,  the  addition  of  fluorine  atoms  further  weakens  the  C=C  bond  of  the  PVA 
molecule  and  it  creates  a  saturated  radical  prone  to  chain  scission.  The  fluorine  gases  combine 
with  the  carbon  and  oxygen  atoms  and  create  a  fluorinated  ethane  and  ethylene  derivative. 
These  stable  fluorine  products  remain  on  the  surface  of  the  PVA.  If  the  concentration  of  the 
fluorine  atoms  is  increased  beyond  a  threshold,  fluorinated  ethane  will  have  retarding  effects 
on  the  etching  rate.  The  fluorinated  ethane  has  to  be  removed  before  reaching  the  PVA 
surface  and  etching  the  PVA  surface.  Hence,  the  etching  rate  exhibits  a  maximum  for  a  given 
concentration  ratio  of  02  and  CF4  and  it  rapidly  decreases  with  the  increase  or  decrease  of  the 
CF4  concentration. 

We  have  performed  experiments  where  the  concentration  of  CF4  was  varied  in  order  to 
determine  the  optimum  etching  rate.  From  Fig.  3,  we  can  observe  that  for  30%  of  CF4  in  a 
total  mixture  of  02  and  CF4,  the  PVA  exhibits  a  maximum  etching  rate.  Deviations  from  this 
optimal  concentration  have  retarding  effects  on  the  etching  rate  as  it  is  expected  form  the 
theory.  Since  the  SU-8  photoresist  is  a  polymer,  it  also  exhibits  similar  etching  behavior  as  the 
PVA.  Due  to  its  different  molecular  composition,  the  SU-8  has  slightly  higher  etching  rate 
then  the  PVA.  This  should  be  taken  into  account  for  the  final  thickness  of  the  SU-8,  as  the 
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SU-8  acts  as  protective  layer  for  the  PVA  and  it  should  remain  until  the  unprotected  PVA  is 
etched  completely.  From  the  experimental  data,  an  Oxygen-to-Freon  (CF4)  ratio  of  3:1  yields 
an  optimum  etching  results.  For  this  gases  concentration,  the  etching  rate  of  the  SU-8  is 
0.75pm/min,  while  the  etching  rate  for  the  PVA  is  0.5pm/min.  High  etching  rate  and  low 
selectivity  was  chosen  in  order  to  minimize  the  etching  time  of  the  PVA  layer  and  hence 
minimize  the  under  etching. 


Fig.  3.  Etching  rate  of  PVA  and  SU-8  as  a  function  of  a  CF4  concentration  in  an  02  and 
CF4  gas  mix. 

5.  Dual-tier  micropolarization  array:  Experimental  results 

The  final  two-tier  micropolarization  array  was  tested  for  its  structural  and  optical  properties. 
A  scanning  electron  microscope  (SEM)  was  used  to  evaluate  the  photoresist  and  the  etched 
PVA  structures.  A  comparison  of  the  size  of  both  structures  indicated  the  amount  of  under¬ 
etching  due  to  the  selective  RIE  procedure. 

In  Fig.  4-a  and  Fig.  4-b,  the  SU-8  square  structures  obtained  after  the  photolithography 
procedure  (see  Fig.  2-h)  are  presented.  Fig.  4-a  presents  the  photoresist  structures  viewed 
from  the  top,  while  Fig.  4-b  present  an  array  of  photoresist  pillars  recorded  under  52  degree 
angle  tilt.  The  photoresist  squares  are  10pm  by  10pm  wide  and  15pm  tall,  with  10pm  spacing 
between  neighboring  structures.  The  corners  of  the  square  structures  are  slightly  rounded  due 
to  edge  diffraction  effects  from  the  mask.  The  rounding  effects  are  minimized  by  optimizing 
pre-baking,  post-baking  and  exposure  time  of  the  photoresist.  The  uniform  periodic  structures 
are  observed  in  Fig.  4-b. 
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Fig.  4.  Scanning  electron  microscope  images  of  the  developed  SU-8  photoresist  structures. 
Left  panel  shows  a  top  view  of  a  small  neighborhood  of  SU-8  squares  with  10pm  by  10pm 
pitch.  Right  image  shows  an  array  of  periodic  SU-8  square  structures  with  15 jam  thickness, 
which  was  recorded  under  52°  angle  tilt. 


After  the  selective  reactive  ion  etching  of  the  top  PVA  layer  (see  Fig.  2-n),  the 
micropolarization  structures  were  evaluated  under  an  SEM.  Fig.  5-a  and  Fig.  5-b  present  the 
top  PVA  layer  of  the  complete  two-tier  filter.  The  left  panel  presents  a  top  view  of  a  small 
neighborhood  of  micropolarizer  structures.  The  square  PVA  structures  have  8pm  by  8pm  size 
with  12pm  spacing  between  neighboring  structures.  Therefore,  the  undercutting  due  to  the 
RIE  is  1pm  per  side  and  the  final  size  of  the  PVA  square  structures  has  decreased  by  2pm 
from  the  original  10pm  photoresist  structures.  Readjusting  the  size  of  the  squares  on  the  mask 
(i.e.  photoresist  structure)  to  12pm  will  lead  to  10pm  PVA  square  structures.  Since,  our  photo 
pixel  has  a  fill  factor  of  64%  (i.e.  the  photodiode  area  is  8pm  by  8pm),  the  obtained  size  of 
the  PVA  structures  satisfied  the  minimum  size  requirements. 

The  right  image  in  Fig.  5,  which  was  recorded  under  52  degree  angle  tilt,  presents  a  single 
PVA  micro  structure  of  size  8pm  by  8pm  and  10pm  thickness.  The  PVA  structure  does  not 
contain  any  residual  SU-8  photoresist  since  the  thickness  of  the  photoresist  was  optimized  to 
be  completely  etched  away  during  the  selective  RIE  procedure.  Furthermore,  we  can  observe 
that  the  top  of  the  PVA  structure  is  not  smooth,  which  indicates  that  it  has  been  partly  etched. 
This  is  a  result  of  small  variations  in  the  PVA  thickness  as  well  as  variations  in  the  SU-8 
photoresist.  Hence,  precise  etching  time  for  the  PVA  can  not  be  determined.  The  top  of  the 
PVA  structures  was  spin  coated  with  an  adhesive  promoter  AD420  since  it  has  an  index  of 
refraction  similar  to  the  PVA  (n=1.41).  This  prevented  any  undesirable  diffraction  which 
might  occur  at  the  surface  of  the  PVA.  Although  the  top  portion  of  the  PVA  has  been  etched 
(around  0.3pm  from  the  10  pm  total  thickness),  the  optical  measurements  of  the  PVA 
structures  have  shown  that  the  extinction  ratios  have  been  preserved  during  the  etching 
procedure. 
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(a)  (b) 


Fig.  5.  Scanning  electron  microscope  images  of  the  top  PVA  tier.  Left  image  shows  an 
array  of  8pm  by  8pm  PVA  structures.  Right  panel  shows  a  single  PVA  structure  with  8pm 
pitch  and  10pm  thickness,  which  was  recorded  under  52  degree  angle  tilt. 


The  optical  properties  of  the  dual-tier  micropolarization  filter  were  evaluated  under  an 
optical  microscope  via  back  polarized  illumination  of  the  sample  with  three  different 
wavelengths:  720nm  wavelength  (red  light),  580nm  wavelength  (green  light)  and  480nm 
wavelength  (blue  light).  We  have  created  two  samples  of  a  dual- tier  micropolarization  filter. 
In  the  first  sample  the  two  layers  of  micropolarizers  are  offset  by  90  degrees,  while  in  the 
second  sample  they  are  offset  by  45  degrees.  The  second  sample  contains  the  desired  pattern 
which  will  be  integrated  with  a  CMOS  image  sensor  in  order  to  extract  the  first  three  Stokes 
parameters.  The  first  sample  was  used  for  measurement  purposes  and  for  visual  clarity, 
although  it  can  be  used  for  a  polarization  difference  (contrast)  computation  [4,9,10]. 

The  optical  characteristics  of  the  dual  axial  micropolarizers  offset  by  90  degrees  are 
presented  in  Fig.  6.  In  Fig.  6-a,  the  sample  was  illuminated  with  polarized  light  having 
polarization  perpendicular  (parallel)  to  the  transmission  axis  (axis  of  polarization)  of  the  PVA 
structures  in  the  first  tier.  The  square  structures  in  the  first  tier  appear  bright  in  intensity  and 
little  darker  compared  to  the  background  since  the  PVA  attenuates  part  of  the  parallel 
polarized  light  (PVA  transmission  of  about  40%  has  been  measured  and  reported  in  the 
literature  [26]).  The  square  structures  in  the  second  tier  appear  opaque.  They  attenuate  the 
intensity  of  the  incoming  light  waves  since  their  axis  of  polarization  is  oriented  perpendicular 
to  the  polarization  of  the  incident  polarized  light.  In  Fig.  6-b  the  sample  was  illuminated  with 
polarized  light  perpendicular  (parallel)  to  the  transmission  axis  (axis  of  polarization)  of  the 


(a)  (b) 


Fig.  6.  Experimental  results  for  the  dual-tier  micropolarization  filters  offset  by  90  degrees.  The 
first  layer  in  the  left  sample  attenuates  perpendicular  polarized  light  while  the  second  layer  is 
transparent.  The  reverse  is  observed  in  the  right  sample. 
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PVA  structures  in  the  second  tier.  The  reverse  effects  are  observed  in  this  image,  where  the 
top  layer  is  opaque  and  the  bottom  layer  is  transparent  due  to  the  90  degree  polarization  offset 
of  the  illuminating  light  source. 

Since  the  square  structures  in  the  dual  axial  filter  reside  in  different  tiers  separated  by 
10pm  and  the  images  were  taken  under  a  microscope  with  5  pm  depth  of  focus,  the  contours 
of  the  square  structures  in  the  bottom  tier  appear  slightly  blurred.  Refocusing  the  microscope 
to  image  the  bottom  tier  structures  verified  the  sharpness  of  the  squarely-etched  structures. 
Since  the  final  thickness  of  the  dual  axial  filter  is  ~20pm,  the  angle  of  incidence  of  the 
incoming  light  will  be  limited.  For  example,  for  large  angles  of  incidence,  the  incoming  light 
will  be  filtered  with  one  micro  filter  and  registered  by  a  neighboring  pixel  on  the  CMOS 
image  sensor.  Using  thinner  layers  will  reduce  this  effect.  In  addition,  introducing  micro 
lenses  on  top  of  the  micropolarization  array  in  the  future  will  circumvent  this  problem. 

The  optical  characteristics  of  the  dual-axial  micropolarizers  offset  by  45  degrees  are 
presented  in  Fig.  7.  The  images  in  Fig.  7 -a,  7-b,  7-c  and  7-d  are  recorded  with  0,  45,  90  and 
135  degrees  of  polarized  light,  respectively.  In  Fig.  7-a  (Fig.  7-c),  the  sample  was  illuminated 
with  polarized  light  with  polarization  parallel  (perpendicular)  to  the  transmission  axis  of  the 
PVA  structures  in  the  first  tier  i.e.  with  0  (90)  degree  polarized  light.  Hence,  the  square 
structures  in  the  first  tier  appear  opaque  in  intensity  in  Fig.  7-a  and  transparent  in  Fig.  7-c. 
Since  the  second  tier  is  offset  by  45  degrees,  it  is  difficult  to  visualize  its  relative  offset.  The 
polarization  properties  of  the  second  tier  are  clearly  evident  in  Fig.  7-b  and  Fig.  7-d.  In  Fig.  7- 
b  (Fig.  7-d),  the  sample  was  illuminated  with  polarized  light  parallel  (perpendicular)  to  the 
transmission  axis  of  the  PVA  structures  in  the  second  tier  i.e.  with  45  (135)  degree  polarized 
light.  Hence,  the  square  structures  in  the  second  tier  appear  opaque  in  intensity  in  Fig.  7-b  and 
transparent  in  Fig.  7-d. 


Tier  1  Tier  1 


Tier  2  Tier  2 

(a)  (b)  (c)  (d) 

Fig.  7.  Experimental  results  for  the  two  axial  micropolarization  filters  offset  by  45  degrees. 

The  images  are  illuminated  with  0,  45,  90  and  135  degree  polarization  for  the  incident 
polarized  light,  respectively. 

Figure  8  and  Fig.  9  present  movies,  demonstrating  the  polarization  filtering  capabilities  of 
the  dual-tier  and  single-tier  micro-structured  PVA,  respectively.  The  offset  of  the  two-tier 
samples  is  45  degrees.  The  movie  files  are  composed  of  36  frames,  where  the  angle  of 
polarization  of  the  light  source  is  varied  between  0  degrees  and  180  degrees  in  increments  of 
5  degrees  between  frames.  The  filter  was  illuminated  with  red-light  wavelength  in  both 
examples;  hence  the  transparent  background  of  the  sample  appears  red.  The  samples  were 
illuminated  with  all  three  wavelengths  separately  in  order  to  evaluate  the  dependence  of  the 
extinction  ratios  on  the  wavelength,  but  for  the  sake  of  brevity  only  one  wavelength  is 
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Fig.  8.  (3  MB)  Movie  of  the  dual 
axial  micropolarizer  array  (13MB 
version). 


Fig.  9.  (3  MB)  Movie  of  a  single 
micropolarizer  element  (12MB 
version). 


presented  here  in  the  movie  files.  In  Fig.  8,  it  can  be  observed  that  initially  the  bottom  layer  is 
transparent  and  the  top  layer  is  opaque.  As  the  angle  of  polarization  is  varied  between  frames, 
the  transparency  shifts  from  the  top  PVA  tier  to  the  bottom  PVA  tier.  Since  the  data  was  taken 
while  focusing  on  the  PVA  structures  on  the  top  tier,  the  PVA  structures  in  the  bottom  tier 
appear  slightly  blurred  and  unfocused.  In  Fig.  9,  we  observe  the  polarization  behavior  of  a 
single  PVA  tier.  The  PVA  structure  gradually  changes  from  transparent  to  opaque  as  the  angle 
of  polarization  is  changed.  In  this  Fig.,  we  can  closely  monitor  that  the  effective  size  of  the 
PVA  structures  is  8pm  by  8pm,  demonstrating  anisotropic  under-etching  of  1pm  during  the 
RIE  procedure. 

The  two-tier  filter,  where  the  tiers  were  offset  by  90  degrees,  was  also  recorded  with  a  12 
bit  grayscale  camera.  Since  the  extinction  ratio  of  the  un-pattemed  PVA  is  around  1000  [26], 
the  sensitivity  of  the  imaging  devices  has  to  be  at  least  10  bits  or  higher.  Two  regions  of  10  by 
10  pixels  were  selected  in  Fig.  8,  which  corresponded  to  two  PVA  structures  in  separate  tiers. 
The  average  intensity  value  was  computed  and  normalized  to  the  incident  light  intensity  on 
the  filter.  The  data  was  recorded  over  36  frames,  where  the  angle  of  polarization  was 
incremented  by  5  degrees  between  frames.  The  transmission  properties  of  both  tiers, 
presented  in  Fig.  10,  closely  follow  Malus’  cosine  low  for  polarization  irradiance  [1].  Since 
the  two  PVA  layers  were  offset  by  90  degrees,  the  maximum  and  minimum  transmissions 
between  the  two-tiers  are  expectedly  shifted  by  90  degrees. 
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Polarization  angle  of  light  source 


Fig.  10.  Transmission  of  both  PVA  structures  follow  Malus’  law  for  polarization  irradiance 


The  extinction  ratios  for  the  patterned  PVA  structures  were  also  evaluated  at  three 
different  wavelengths.  Fig.  llFigure  11  presents  the  transmission  percentage  as  a  function  of 
the  angle  of  polarization  for  three  different  wavelengths.  Also,  transmission  data  for  an 
unpatterned  PVA  sample  was  recorded  and  presented  with  black  dotted  line  in  Fig.  11.  From 
this  data,  the  extinction  ratios,  defined  as  the  ratio  of  the  maximum  to  minimum  transmission 
for  a  given  wavelength,  were  calculated.  The  maximum  transmissions  for  red  and  blue 
wavelength  were  recorded  to  be  45%  and  40%  respectively.  The  minimum  transmissions  for 
the  red  and  blue  wavelength  were  recorded  to  be  0.4%  and  0.04%  respectively.  Therefore,  the 
extinction  ratio  for  blue  and  green  wavelengths  is  -1000  and  for  red  wavelengths  the 
extinction  ratio  is  -100.  The  extinction  ratios  for  unpatterned  PVA  evaluated  with  red 
wavelength  is  -100.  The  data  in  Fig.  11  demonstrates  that  the  polarization  properties  of  the 
micropolarization  PVA  structures,  created  using  selective  RIE  are  not  altered  and  they  are 
similar  to  the  polarization  properties  of  an  unpatterned  and  an  un-etched  PVA  filter. 
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Fig.  11.  Transmission  of  a  single  PVA  micro  structure  under  different  wavelengths.  The 
transmission  properties  of  unpatterned  PVA  are  also  presented. 


6.  Conclusion 

An  outline  of  a  novel  polarization  filtering  architecture  for  real  time  polarimetric  imaging  has 
been  described.  This  sensory  system  requires  accurate  patterning  of  thin  film  polarizers.  We 
have  outlined  the  methodology  necessary  to  pattern  and  etch  lOpm-thick  PVA  polarizer  in 
order  to  create  8pm  by  8pm  square  structures.  These  methods  would  allow  patterning  of 
polarization  micro  structures  necessary  to  be  placed  on  top  of  an  imaging  sensor.  The 
extinction  ratios  of  the  polarization  filter  are  around  1000  for  blue  and  green  wavelength  and 
100  for  red  wavelengths.  In  our  next  phase  of  research  efforts,  the  final  array  of 
micropolarizer  will  be  mounted  on  top  of  a  custom  made  image  sensor  for  real  time 
polarimetric  computation. 
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